diff --git a/tests/tasks/uipath-troubleshoot/CLAUDE.md b/tests/tasks/uipath-troubleshoot/CLAUDE.md index 0d1f101ca6..64124a802d 100644 --- a/tests/tasks/uipath-troubleshoot/CLAUDE.md +++ b/tests/tasks/uipath-troubleshoot/CLAUDE.md @@ -13,7 +13,7 @@ Trigger: Skip: -- Tweaking an existing scenario's manifest or fixtures — edit them directly, no generation pass needed. +- Tweaking an existing scenario's protected fixture — edit it directly, no generation pass needed. - Adding tests for *other* skills — use `.claude/commands/generate-tasks.md` instead. ## Inputs @@ -27,7 +27,7 @@ A new scenario needs four sources. Three are mandatory; the fourth is optional. | `--transcript ` | yes | A `.jsonl` file or a directory. Directory mode walks `*.jsonl` recursively and treats files under `subagents/` as sub-agent transcripts (their `uip` calls count, their final text is ignored). Source of truth for `uip` calls + presenter output. | | `--resolution ` | no | Pre-written `RESOLUTION.md`. If omitted, the generator extracts the presenter's final assistant message from the transcript. | | `--scenario-name ` | no | Folder name for the new scenario. If omitted, inferred from project name + failing job key. | -| `--group ` | yes | Group folder — e.g. `activity-packages`, `products/orchestrator`, `runtime-exceptions`, `cross-system`. Sets placement, the depth-correct `_shared` path, and the default domain tag. Activity packages are flat under `activity-packages/`; the `--scenario-name` carries the package token (`db-`, `word-`, …). See [Scenario grouping](#scenario-grouping). | +| `--group ` | yes | Group folder — e.g. `activity-packages`, `products/orchestrator`, `runtime-exceptions`, `cross-system`. Sets placement and the default domain tag. Activity packages are flat under `activity-packages/`; the `--scenario-name` carries the package token (`db-`, `word-`, …). See [Scenario grouping](#scenario-grouping). | ## Workflow — ask before writing @@ -46,16 +46,16 @@ python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ --dry-run ``` -`--group` is the group folder the scenario belongs to (see [Scenario grouping](#scenario-grouping)). It sets three things at once: where the scenario folder is written, the depth-correct `_shared` path inside `task.yaml`, and the default product/domain tag. Pass it for every new scenario — omitting it falls back to flat placement at the suite root (deprecated). +`--group` is the group folder the scenario belongs to (see [Scenario grouping](#scenario-grouping)). It sets placement and the default product/domain tag. Pass it for every new scenario — omitting it falls back to flat placement at the suite root (deprecated). -Output: a plan describing every file that would be written (paths + sizes + first lines), the inferred scenario name, the manifest rules extracted, and any scrub substitutions that would be applied. +Output: a plan describing every file that would be written (paths + sizes + first lines), the inferred scenario name, the recorded protected-fixture responses extracted, and any scrub substitutions that would be applied. ### 2. Confirm with the user Show the preview. Ask: - Is the scenario name correct? -- Are the manifest rules complete (every `uip` arg signature the agent ran)? +- Are the protected-fixture commands complete (every `uip` argv signature the agent ran)? - Does the extracted `RESOLUTION.md` match the agent's final answer? (Or should they supply one explicitly?) - Are the scrub substitutions correct (placeholders mapped to the right real values)? @@ -75,15 +75,15 @@ Writes the scenario folder. Prints the path. Exits. After write: -1. Run the smoke test pattern to confirm the mock dispatcher resolves: +1. Run the scenario to confirm the protected mock service resolves: ```bash cd tests .venv/bin/coder-eval run tasks/uipath-troubleshoot///task.yaml -e experiments/default.yaml -v ``` 2. The first run should score 1.0 — the test was generated from a known-good resolution. -3. Confirm every expected call was hit from the run artifact's call log (sealed runs encode each record): +3. Confirm every expected call was hit from the run artifact's call log: ```bash - python tests/tasks/uipath-troubleshoot/_shared/scripts/coverage_report.py --dump /m/.log + python tests/tasks/uipath-troubleshoot/_shared/scripts/coverage_report.py /default/ ``` ## Mandatory scrub list @@ -110,11 +110,11 @@ Scenarios sit in group folders. Do NOT add a scenario at the flat suite root — ``` tests/tasks/uipath-troubleshoot/ -├── _shared/ # shared mock dispatcher + scripts (never a scenario) +├── _shared/ # scenario authoring and validation scripts ├── smoke-manifest-commands/ # the sole smoke task (stays at root) ├── activity-packages/ # FLAT — scenarios named - -│ ├── db-execute-query-timeout-expired/ data/m/r/ … -│ ├── excel-rr-sheet-bytes/ data/m/r/ … +│ ├── db-execute-query-timeout-expired/ data/uip-fixture.json +│ ├── excel-rr-sheet-bytes/ data/uip-fixture.json │ ├── uia-node-not-found/ word-replace-text-file-locked/ … ├── products/ │ ├── orchestrator/ / … @@ -137,7 +137,7 @@ tests/tasks/uipath-troubleshoot/ | Generic .NET workflow exception (null-ref, argument-null) not tied to a package | `runtime-exceptions` | — | | Root cause genuinely spans ≥2 systems (e.g. an Excel activity failing on an IS connection) | `cross-system` | — | -The `--group` flag wires up placement, the depth-correct `_shared` path, and the default tag. The `_shared` path depth follows the nesting: `activity-packages//`, `runtime-exceptions//`, and `cross-system//` are one level deep → `../../_shared/mock_template`; `products///` is two deep → `../../../_shared/mock_template`. For activity packages pass `--group activity-packages` and a `--scenario-name` that already carries the package token (e.g. `db-execute-query-timeout-expired`). +The `--group` flag wires up placement and the default tag. For activity packages pass `--group activity-packages` and a `--scenario-name` that already carries the package token (e.g. `db-execute-query-timeout-expired`). ## Scenario folder layout @@ -145,35 +145,57 @@ A single scenario (leaf) under its group folder: ``` tests/tasks/uipath-troubleshoot/// -├── task.yaml # tags, mock_path_dirs, llm_judge criteria +├── task.yaml # tags, protected_mocks, llm_judge criteria ├── RESOLUTION.md # ground truth for the LLM judge -├── data/ # short dir names keep Windows paths under MAX_PATH (260) -│ └── m/ -│ └── r/ -│ ├── README.md # what the original session uncovered — deleted by `m/seal` -│ ├── manifest.json # rules (canned + passthrough) + unmocked_default -│ └── *.json # canned stdout per rule with `file:` (name = sha1[:10] of args) +├── README.md # maintainer description; not staged to the agent +├── data/ +│ └── uip-fixture.json # private recorded argv -> response map └── process/ # snapshot of the failing UiPath project (optional) └── ... ``` -`task.yaml` MUST set `sandbox.mock_path_dirs: ["m"]` and overlay the scenario's `data/` via a `template_dir` source — without it, bare `uip` resolves to the real CLI and the test will try to authenticate. (Dir names are single letters — `data/m/r/` — deliberately: verbose `fixtures/mocks/responses/` paths pushed the plugin's installed file paths past the Windows 260-char `MAX_PATH` limit.) +`task.yaml` MUST declare a `sandbox.protected_mocks` entry for `uip` and MUST NOT pin `sandbox.driver` (the experiment config decides; protected mocks are driver-independent). Do not add `data/` as a template source: coder-eval serves `data/uip-fixture.json` from a host-side mock service outside the agent workspace and places only a thin `uip` client shim in the workspace. -## Mock dispatch precedence +## Protected mock contract -The shared `m/uip` dispatcher walks the manifest's `rules` array (first match wins). Matching is token-aware (order-independent, quotes stripped, `--flag=value` split, `--output ` ignored) with a substring fallback — see the module docstring in `_shared/mock_src/uip.py` (the dispatcher's source; the template ships it as a compressed docstring-stripped blob). Authoring consequences: list specific rules before generic ones (a rule with fewer tokens matches a superset of invocations), and prefer dropping volatile flags (e.g. `-f` vs `--folder-key`) from match strings when the remaining tokens are unique. Shadow audit: for rules i < j in one manifest, if tokens(match_i) ⊆ tokens(match_j) and they serve different files/exit codes, rule j is unreachable for its canonical invocation. +Each response in `data/uip-fixture.json` contains `argv`, `exit_code`, and optional `stdout`/`stderr`. Troubleshoot fixtures use `match_mode: subset`: a rule matches when every rule token appears among the invocation's tokens. Token comparison is order-independent, `--flag=value` and `--flag value` are equivalent, and `--output ` is ignored on both sides. The agent may therefore add benign extra flags (`--output-filter`, `--limit`, `--level`, `--folder-key`) and still receive the recorded response — this mirrors the run-observed behavior of real investigations. An invocation matching no rule receives the fixed `default` response. -Each rule has one of: +Subset dispatch is first-match-wins in fixture-file order, so generated fixtures sort rules most-specific-first (token count descending). Never hand-append a generic rule above a more specific one. -- `file: ` — return the canned response under `r/`. -- `passthrough: true` — proxy to the real `uip` CLI installed on the host. Use this for open-ended commands like `docsai ask` whose query strings vary between runs. Responses are cached to the sandbox's `r/_cache/.json` for in-run reuse; the cache is **not** persisted to the source — every run hits the live CLI on its first call for each unique query. +```json +{ + "version": 1, + "responses": [ + { + "argv": ["or", "jobs", "get", ""], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\"Result\":\"Success\"}\n" + } + ], + "default": {"exit_code": 0, "stdout": "[]\n"} +} +``` -When no rule matches: +Outputs, request size, response size, request count, and passthrough duration are bounded. The mock protocol has no file-read or fixture-dump operation. -1. `unmocked_default` (if set) — return its `response` + `exit_code`. -2. Otherwise, error on stderr. +The scenario's sandbox block has this shape: -Test runs require valid `uip` auth on the host (set via `.env` or environment) for any rule with `passthrough: true` to succeed. +```yaml +sandbox: + python: {} + template_sources: + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] +``` + +`template_sources` is omitted when the scenario has no `process/` snapshot. ## Task YAML requirements @@ -224,7 +246,7 @@ Every faithful-replay scenario is an end-to-end investigation — tag it **`e2e` The PR smoke gate (`.github/workflows/smoke-skills.yml`) treats **any** change under `skills/uipath-troubleshoot/` as a skill-source change and runs that skill's **entire `smoke`-tagged set**. A `smoke`-tagged scenario therefore runs on every docs/playbook PR — each scenario is a multi-minute agent run, so the gate becomes slow, expensive, and exposes unrelated PRs to scenario flakiness and CI-infra blips (image-pull, judge variance). That is the wrong signal for a fast PR gate. -**The ONLY troubleshoot task tagged `smoke` is [`smoke-manifest-commands`](./smoke-manifest-commands/task.yaml)** — a fast, deterministic fixture/manifest-command validation (no agent investigation). It is the sole troubleshoot smoke-gate check by design. +**The ONLY troubleshoot task tagged `smoke` is [`smoke-manifest-commands`](./smoke-manifest-commands/task.yaml)** — a fast, deterministic protected-fixture command validation (no agent investigation). It is the sole troubleshoot smoke-gate check by design. Its historical directory name is retained to avoid changing the task identity. Rules: @@ -236,39 +258,11 @@ Rules: The skill writes investigation artifacts to `.local/investigations/` — NOT `.investigations/`. Every `file_exists` criterion path and every post-run script path MUST use `.local/investigations/...`. -### `pre_run` — seal the mock store - -Every scenario MUST seal its fixtures before the agent starts: - -```yaml -pre_run: - - command: "python m/seal" - timeout: 60 - fail_on_error: true -``` - -`data/m/r/` is staged into the agent's working directory so the `m/uip` shim can resolve it — which also lets the agent `cat ./m/r/*.json` and read the recorded `uip` outputs, reaching the root cause without invoking `uip` or the skill at all. `m/seal` packs the manifest + every fixture into an opaque `m/.store` (zlib+base64) and deletes `r/`; the shim reads `.store` transparently. After sealing there is no readable fixture in the sandbox. - -The mock machinery itself is never readable in the sandbox — its source would otherwise tell the agent how the test evidence is kept (the dispatcher's docstring documents the manifest schema, the `.store` format, and the annotation-stripping pass). The sources live in `_shared/mock_src/` (`uip.py`, `seal.py`) and are NOT staged; the template ships them as compressed docstring-stripped blobs (`m/.uip.bin`, `m/.seal.bin` — AST-stripped source, zlib+base64, packed by `_shared/scripts/compile_mocks.py`) plus thin loaders at `m/uip` and `m/seal` that decode and exec the blob in memory. A blob shows a single base64 run to `cat`/`strings` (unlike bytecode, which keeps every string literal readable) and runs on ANY host CPython >= 3.10 — no version matrix, no PATH setup for local runs (the sandbox resolves `python` from the host PATH). After editing a source, repack with `uv run --python 3.13 tests/tasks/uipath-troubleshoot/_shared/scripts/compile_mocks.py` (3.13 keeps the output byte-stable) and commit the regenerated blobs. The dispatcher's call log (`m/.log`) is written zlib+base64-encoded on sealed runs — plain records name the matched rule and fixture file, proof the CLI is mocked. Decode it with `coverage_report.py --dump`. - -`fail_on_error: true` is deliberate — a silent seal failure restores the leak. `m/seal` is idempotent and no-ops when there is no `r/manifest.json`, so a re-run in a reused sandbox still exits 0. - -The passthrough cache moves to `m/_cache` (beside the shim) so `docsai` proxying keeps working after `r/` is gone. - -`_build_task_yaml` in `generate_scenario.py` emits this block, so generated scenarios get it automatically. Hand-written scenarios MUST add it. - -Sealing also removes the scenario's `README.md`, which lives at `data/m/r/README.md` for exactly that reason — `seal` deletes the whole `r/` directory, so the write-up describing the root cause never reaches the agent even though `data/` is staged. Keep it there; a README at the task root sits in the task dir that coder-eval bind-mounts read-only into the container (`$TASK_DIR`), and one at `data/` root lands in the agent's working directory unsealed. - ### `docsai` mocking -Any rule matching `uip docsai ask ...` in `manifest.json` MUST be `passthrough: true`. Query strings vary between runs and canned responses go stale immediately; the dispatcher caches passthrough responses per query for in-run reuse. +Open-ended `uip docsai ask ...` calls use the narrowly typed passthrough prefix shown above. Query strings vary between runs and canned responses go stale. The isolated mock service invokes the real `uip` executable without a shell, caches each exact argv response in memory for the run, and never reveals the executable path to the agent. Test runs require valid host `uip` auth. -```json -{ - "match": "docsai ask", - "passthrough": true -} -``` +Do not configure broad passthrough prefixes such as `[or]`, `[auth]`, or `[tools]`. Recorded tenant and job data always stays fixture-backed. ### `success_criteria` @@ -371,7 +365,7 @@ success_criteria: ## Anti-patterns - **Do not** add `command_executed` criteria. The lean contract is `skill_triggered + llm_judge` only — no `command_executed`, no `file_exists`, no `file_contains`. Asserting a specific CLI command was run encodes one investigation path and false-fails agents that reach the correct conclusion via a different (still valid) route. If a test needs to verify a specific CLI fact, put the fact in `RESOLUTION.md` and let the judge grade it. See the `success_criteria` section above for the full forbidden list. -- **Do not** hand-edit a generated scenario's `manifest.json` to "make tests pass." If the agent calls a command not in the manifest, that's a coverage gap — add a rule with the verbatim recorded response. +- **Do not** broaden a response argv to "make tests pass." If the agent calls an unconfigured command, add a response rule with the verbatim recorded stdout or keep the fixed default when empty exploration is intentional. - **Do not** include the original `.local/investigations/` outputs in the committed scenario. The fresh run produces its own. - **Do not** ship real email addresses, real personal Windows paths, or real machine hostnames. The scrub pass is mandatory. - **Do not** keep names that encode the diagnosis. The agent-visible surfaces — process / project / release names, workflow / `.xaml` filenames, `x:Class`, activity `DisplayName`, and the `initial_prompt` — MUST NOT broadcast the failure. Real sessions routinely carry such names (`ERN_O365_MailFolderNotFound`, activity `"Get Mail from nonexistent folder (expect ArgumentNullException)"`); left in, the agent passes by reading the name instead of investigating — or never invokes the skill at all. Neutralize them to realistic business names during generation, exactly like the PII scrub, keeping the fault ONLY in the retrieved log/exception text. A user pasting the *symptom* into the prompt (`faulted with System.IO.FileNotFoundException`) is fine; a *name* that states the cause is not. diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_src/seal.py b/tests/tasks/uipath-troubleshoot/_shared/mock_src/seal.py deleted file mode 100644 index f9fa65e82f..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_src/seal.py +++ /dev/null @@ -1,91 +0,0 @@ -#!/usr/bin/env python3 -"""Seal the mock fixture store so the agent cannot read the recorded evidence. - -Runs ONCE per task, in `pre_run` (before the agent starts). It packs the -manifest + every response fixture under `r/` into a single opaque blob -`/.store`, then deletes the `r/` directory entirely. - -Why: the mock backing store (`m/r/*.json`) is staged into the agent's working -directory so the `m/uip` shim can resolve it. But that also lets the agent -`cat ./m/r/*.json` and read the pre-recorded `uip` outputs directly — including -the diagnosis-revealing log lines — diagnosing WITHOUT ever invoking `uip` or -the `uipath-troubleshoot` skill (empirically the dominant `skill_triggered` -failure mode). After sealing there is no readable fixture in the sandbox: the -`r/` directory is gone and `.store` is opaque (zlib+base64). The shim -(`m/uip`) transparently reads `.store` instead of `r/`. - -This script ships to the sandbox only as a compressed docstring-stripped -blob (`m/seal` is a thin loader for `m/.seal.bin`, packed by -`_shared/scripts/compile_mocks.py`), so nothing readable in the sandbox -documents the manifest schema or the `.store` format. - -Idempotent and safe to run anywhere: - - No `r/manifest.json` present → no-op (exit 0). Lets an experiment-level - pre_run run this for EVERY task; non-mock tasks simply skip, and a - re-run in a reused sandbox (where sealing already removed `r/`) is a - no-op. - - A PARTIAL seal (a crash mid-way) always leaves `r/manifest.json` in - place (the store write and `rmtree` happen last), so the pre_run retry - RESUMES the seal — every step is idempotent or skip-guarded. `.store` - alone is never treated as proof the seal completed. - -Blob format (`.store`): base64( zlib( utf-8 json( { - "manifest": , - "files": { "": "", ... } -} ) ) ). Raw bytes are preserved per file so UTF-16/BOM fixtures survive. - -The passthrough cache (`r/_cache`, used only by `docsai ask` proxy rules) is -moved to `/_cache` so live-proxy caching keeps working after `r/` -is removed; it holds docs Q&A, not scenario evidence. -""" - -import base64 -import json -import shutil -import sys -import zlib -from pathlib import Path - -# Sandboxes execute this file as a compressed blob (`m/.seal.bin`, decoded -# and exec'd by the `m/seal` stub with __file__ set to the blob's path in the -# mock dir), so every data path anchors correctly there and when running this -# source directly. -SCRIPT_DIR = Path(__file__).resolve().parent -RESPONSES_DIR = SCRIPT_DIR / "r" -MANIFEST_PATH = RESPONSES_DIR / "manifest.json" -STORE_PATH = SCRIPT_DIR / ".store" -CACHE_SRC = RESPONSES_DIR / "_cache" -CACHE_DST = SCRIPT_DIR / "_cache" - - -def main() -> int: - # Nothing to seal: a non-mock task (mock template never staged with - # fixtures), or a reused sandbox where sealing already completed and - # removed `r/`. - if not MANIFEST_PATH.is_file(): - return 0 - - manifest = json.loads(MANIFEST_PATH.read_text(encoding="utf-8")) - - files: dict[str, str] = {} - for item in RESPONSES_DIR.glob("*.json"): - if item.name == "manifest.json": - continue - files[item.name] = base64.b64encode(item.read_bytes()).decode("ascii") - - blob = {"manifest": manifest, "files": files} - packed = base64.b64encode(zlib.compress(json.dumps(blob).encode("utf-8"), 9)) - - # Preserve the passthrough cache (docsai) outside the doomed r/ dir. - if CACHE_SRC.is_dir() and not CACHE_DST.exists(): - shutil.move(str(CACHE_SRC), str(CACHE_DST)) - - # Commit the store, then remove the readable fixture directory. Store - # first: the shim must always find either `.store` or `r/`. - STORE_PATH.write_bytes(packed) - shutil.rmtree(RESPONSES_DIR) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_src/uip.py b/tests/tasks/uipath-troubleshoot/_shared/mock_src/uip.py deleted file mode 100644 index 18afb227b4..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_src/uip.py +++ /dev/null @@ -1,419 +0,0 @@ -#!/usr/bin/env python3 -"""Generic uip mock dispatcher. - -Reads /r/manifest.json and dispatches each invocation to -the first matching rule's response file. All canned data lives in per-scenario -files under r/ (short dir names keep Windows paths under MAX_PATH); this -script is shared across scenarios and never -needs editing when a new scenario is added. - -Every invocation is appended to /.log so post-run analysis can -compare expected vs performed commands and surface exploration the agent did -beyond the manifest's `expected_calls`. On sealed runs each record is written -zlib+base64-encoded (decode with `coverage_report.py --dump`) so the agent -cannot read its own call history and infer the CLI is mocked; unsealed local -runs keep plain JSONL for easy debugging. - -Manifest schema (v2): - { - "version": 2, - "rules": [ - { "match": "or jobs list", "file": "jobs_list.json" }, - { "match": "auth status", "file": "auth_status.json", "exit_code": 0 }, - { "match": "docsai ask", "passthrough": true } - ], - "expected_calls": [ - { "pattern": "or jobs get ", "min": 1, "description": "..." } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0 - } - } - -Each rule has one of: - - `file: ` — return the canned response under r/. - - `passthrough: true` — proxy to the real `uip` CLI installed on the - host. Responses are cached under r/_cache/.json on - first run; subsequent runs replay the cache. Cache files are - committed alongside fixtures so tests stay reproducible offline. - -Dispatch precedence: - 1. First matching rule (first match wins). - 2. `unmocked_default` — if set, return its `response` + `exit_code`. - 3. Otherwise, error on stderr (legacy behavior). - -Matching (see `_rule_matches`) is token-aware, with a plain-substring -fallback. A rule matches an invocation when EITHER: - - a. every whitespace token of `match` is present among the invocation's - tokens (order-independent), OR - b. `match` is a literal substring of `" ".join(argv[1:])` (legacy). - -Tokens are normalized before comparison: surrounding quotes are stripped -(`""` == ``) and `--flag=value` is split into `--flag value`. -This makes rules robust to: - - flag reordering (`--folder-key K --state Faulted` - matches `--state Faulted --folder-key K`) - - extra trailing flags (a `--output json` the rule omits) - - shell-quoted job keys recorded into a manifest match string - (a historical generate_scenario.py quirk). - -List specific patterns before generic ones — first match still wins, and a -generic rule with fewer tokens will match a superset of invocations. A -passthrough rule with `match: "docsai ask"` is the typical way to proxy -open-ended natural-language commands. -""" - -import base64 -import hashlib -import json -import os -import shutil -import subprocess -import sys -import time -import zlib -from pathlib import Path - -# Sandboxes execute this file as a compressed docstring-stripped blob -# (`m/.uip.bin`, decoded and exec'd by the `m/uip` stub with __file__ set to -# the blob's path in the mock dir), so every data path (store, log, cache) -# anchors correctly both there and when running this source directly. -SCRIPT_DIR = Path(__file__).resolve().parent -RESPONSES_DIR = SCRIPT_DIR / "r" -MANIFEST_PATH = RESPONSES_DIR / "manifest.json" -CALL_LOG_PATH = SCRIPT_DIR / ".log" -# Passthrough (docsai) cache. Canonical location is beside the shim so it -# survives `m/seal` removing `r/`; fall back to the legacy `r/_cache` for -# unsealed local runs that shipped committed caches under `r/`. -CACHE_DIR = SCRIPT_DIR / "_cache" -LEGACY_CACHE_DIR = RESPONSES_DIR / "_cache" - -# Sealed fixture store (written by `m/seal`). When present, the manifest and -# every response fixture are read from here — decoded in memory — and the -# readable `r/` directory no longer exists. Opaque on disk (zlib+base64), so -# `cat`-ing it reveals no evidence. `None` until first access; then either the -# decoded dict `{"manifest": ..., "files": {name: rawbytes}}` or the sentinel -# `_NO_STORE` (unsealed run → fall back to `r/`). -STORE_PATH = SCRIPT_DIR / ".store" -_NO_STORE = object() -_STORE: object = None - - -def _get_store(): - """Load and cache the sealed store, or ``_NO_STORE`` if unsealed. - - Decoded once per process. Returns a dict with ``manifest`` (dict) and - ``files`` (name → raw ``bytes``), or ``_NO_STORE`` when there is no - ``.store`` (the shim then reads ``r/`` as before). - """ - global _STORE - if _STORE is not None: - return _STORE - if not STORE_PATH.is_file(): - _STORE = _NO_STORE - return _STORE - blob = json.loads(zlib.decompress(base64.b64decode(STORE_PATH.read_bytes())).decode("utf-8")) - _STORE = { - "manifest": blob["manifest"], - "files": {name: base64.b64decode(b64) for name, b64 in blob.get("files", {}).items()}, - } - return _STORE - - -def _log_call(args: str, rule: dict | None, exit_code: int, error: str | None = None) -> None: - """Append a structured record of this invocation to the call log. - - Best-effort: any logging error is swallowed so a broken log file never - breaks an agent's command. The file is the source of truth for - expected-vs-performed coverage analysis after the run. - - On sealed runs the record is zlib+base64-encoded: the log lives in the - agent's working directory, and plain records (`matched_rule`, `fixture`) - would reveal the CLI is mocked. Unsealed local runs keep plain JSONL. - Decode with `coverage_report.py --dump`. - """ - record = { - "ts": time.time(), - "args": args, - "matched_rule": rule.get("match") if rule else None, - "fixture": rule.get("file") if rule else None, - "exit_code": exit_code, - } - if error: - record["error"] = error - line = json.dumps(record) - if _get_store() is not _NO_STORE: - line = base64.b64encode(zlib.compress(line.encode("utf-8"), 9)).decode("ascii") - try: - with CALL_LOG_PATH.open("a", encoding="utf-8") as f: - f.write(line + "\n") - except OSError: - pass - - -# Pure-formatting flags that never change WHICH resource an invocation -# addresses. Dropped from both the match string and the invocation before -# token comparison, so a rule and an invocation that differ only by an -# `--output json` (present on one side, absent on the other) still match. -# `--output` takes a value (`--output json`), so its following token is -# dropped too. -_NOISE_VALUE_FLAGS = {"--output"} - - -def _tokenize(s: str) -> list[str]: - """Split a command string into normalized tokens. - - Normalization: - - surrounding single/double quotes stripped from each token, so a - manifest match of `or jobs get ""` tokenizes identically to - the shell-stripped invocation `or jobs get `. - - `--flag=value` split into two tokens (`--flag`, `value`) so the - equals-form and space-form of a flag compare equal. - - pure-formatting flags in `_NOISE_VALUE_FLAGS` (and their values) - dropped, so `--output json` never affects matching. - """ - tokens: list[str] = [] - for raw in s.split(): - if raw.startswith("-") and "=" in raw: - flag, _, val = raw.partition("=") - tokens.append(flag.strip("\"'")) - if val: - tokens.append(val.strip("\"'")) - else: - tokens.append(raw.strip("\"'")) - - # Drop noise value-flags and the value token that follows each. - cleaned: list[str] = [] - skip_next = False - for tok in tokens: - if skip_next: - skip_next = False - continue - if tok in _NOISE_VALUE_FLAGS: - skip_next = True - continue - cleaned.append(tok) - return cleaned - - -def _rule_matches(match: str, args: str, arg_tokens: set[str]) -> bool: - """Return True when `match` matches the invocation `args`. - - Token-subset first (order-independent, quote/flag-normalized), then a - plain-substring fallback so any rule that matched under the old - substring dispatcher keeps matching. Additive by construction: this - never matches fewer invocations than the legacy substring test. - """ - if not match: - return False - match_tokens = _tokenize(match) - if match_tokens and all(t in arg_tokens for t in match_tokens): - return True - return match in args - - -def _err(payload: dict, code: int) -> int: - sys.stderr.write(json.dumps(payload) + "\n") - return code - - -def _has_doc_key(node: object) -> bool: - """True when any dict anywhere in `node` carries an `_`-prefixed key.""" - if isinstance(node, dict): - if any(isinstance(k, str) and k.startswith("_") for k in node): - return True - return any(_has_doc_key(v) for v in node.values()) - if isinstance(node, list): - return any(_has_doc_key(v) for v in node) - return False - - -def _prune_doc_keys(node: object) -> object: - """Return `node` with every `_`-prefixed dict key removed, at any depth.""" - if isinstance(node, dict): - return { - k: _prune_doc_keys(v) - for k, v in node.items() - if not (isinstance(k, str) and k.startswith("_")) - } - if isinstance(node, list): - return [_prune_doc_keys(v) for v in node] - return node - - -def _strip_doc_keys(text: str) -> str: - """Drop `_`-prefixed annotation keys (e.g. `_doc`, `_comment`, `_meta`) from a - JSON fixture before it is emitted as `uip` stdout. - - Fixtures carry maintainer annotations that often state the scenario's root - cause. The real CLI never emits such keys, and returning them verbatim would - leak the answer into the agent's evidence. Strip them so the mocked stdout - matches the shape of a real response. - - Recursive on purpose: annotations also appear nested inside the payload (a - `_meta` block hung off `Data`, for instance), which a top-level-only pass - would emit verbatim. - - Only re-serialize when such a key is actually present, so every other fixture - (and any non-JSON body) passes through byte-for-byte unchanged. - """ - try: - doc = json.loads(text) - except (ValueError, TypeError): - return text - if not _has_doc_key(doc): - return text - trailing = "\n" if text.endswith("\n") else "" - return json.dumps(_prune_doc_keys(doc), indent=2, ensure_ascii=False) + trailing - - -def _cache_key(args: str) -> str: - return hashlib.md5(args.encode("utf-8")).hexdigest()[:16] - - -def _load_cache(args: str) -> dict | None: - """Return cached `{stdout, exit_code, args, cached_at}` or None.""" - key = f"{_cache_key(args)}.json" - for path in (CACHE_DIR / key, LEGACY_CACHE_DIR / key): - if not path.is_file(): - continue - try: - return json.loads(path.read_text(encoding="utf-8")) - except (OSError, json.JSONDecodeError): - continue - return None - - -def _save_cache(args: str, stdout: str, exit_code: int) -> None: - """Persist a passthrough response so subsequent runs replay offline.""" - CACHE_DIR.mkdir(parents=True, exist_ok=True) - path = CACHE_DIR / f"{_cache_key(args)}.json" - payload = { - "args": args, - "stdout": stdout, - "exit_code": exit_code, - "cached_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), - } - try: - path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") - except OSError: - pass - - -def _find_real_uip() -> str | None: - """Locate the real `uip` CLI on PATH, excluding the mock's own dir.""" - self_dir = str(SCRIPT_DIR.resolve()) - parts = [] - for d in os.environ.get("PATH", "").split(os.pathsep): - if not d: - continue - try: - if Path(d).resolve() == SCRIPT_DIR.resolve(): - continue - except (OSError, ValueError): - pass - parts.append(d) - return shutil.which("uip", path=os.pathsep.join(parts)) - - -def _passthrough(args: str, original_argv: list[str], rule: dict) -> int: - """Proxy to the real `uip` CLI; cache the response for subsequent runs.""" - cached = _load_cache(args) - if cached is not None: - sys.stdout.write(cached.get("stdout", "")) - exit_code = int(cached.get("exit_code", 0)) - _log_call(args, rule, exit_code, error="passthrough_cached") - return exit_code - - real_uip = _find_real_uip() - if real_uip is None: - _log_call(args, rule, 2, error="passthrough_no_real_uip") - return _err( - {"error": "passthrough requested but real uip not on PATH", "args": args}, - 2, - ) - - proc = subprocess.run( - [real_uip] + original_argv[1:], - capture_output=True, - text=True, - encoding="utf-8", - ) - sys.stdout.write(proc.stdout) - sys.stderr.write(proc.stderr) - _save_cache(args, proc.stdout, proc.returncode) - _log_call(args, rule, proc.returncode, error="passthrough_live") - return proc.returncode - - -def main(argv: list[str]) -> int: - # Force stdout/stderr to UTF-8 so non-ASCII fixture content (e.g. Romanian, - # Cyrillic) doesn't crash on Windows where the default console encoding is - # cp1252. The agent reads stdout via a file redirect, so encoding it as - # UTF-8 is always correct. - if hasattr(sys.stdout, "reconfigure"): - sys.stdout.reconfigure(encoding="utf-8", errors="replace") - if hasattr(sys.stderr, "reconfigure"): - sys.stderr.reconfigure(encoding="utf-8", errors="replace") - - args = " ".join(argv[1:]) - - # Prefer the sealed store (fixtures decoded in memory, `r/` removed); fall - # back to the readable `r/` directory for unsealed local runs. - store = _get_store() - if store is not _NO_STORE: - manifest = store["manifest"] - elif MANIFEST_PATH.is_file(): - manifest = json.loads(MANIFEST_PATH.read_text(encoding="utf-8")) - else: - _log_call(args, None, 2, error="manifest_missing") - return _err({"error": "manifest.json missing", "path": str(MANIFEST_PATH)}, 2) - - # 1. First matching rule wins. - arg_tokens = set(_tokenize(args)) - for rule in manifest.get("rules", []): - if not _rule_matches(rule.get("match", ""), args, arg_tokens): - continue - if rule.get("passthrough"): - return _passthrough(args, argv, rule) - # Fetch the fixture bytes from the sealed store or the `r/` directory. - if store is not _NO_STORE: - raw = store["files"].get(rule["file"]) - if raw is None: - _log_call(args, rule, 2, error="fixture_missing") - return _err({"error": "response file missing", "path": rule["file"]}, 2) - else: - response_file = RESPONSES_DIR / rule["file"] - if not response_file.is_file(): - _log_call(args, rule, 2, error="fixture_missing") - return _err({"error": "response file missing", "path": str(response_file)}, 2) - raw = response_file.read_bytes() - # Tolerate fixtures written by PowerShell (UTF-16 LE BOM) or UTF-8. - if raw.startswith(b"\xff\xfe") or raw.startswith(b"\xfe\xff"): - text = raw.decode("utf-16") - else: - text = raw.decode("utf-8-sig", errors="replace") - text = _strip_doc_keys(text) - sys.stdout.write(text) - exit_code = int(rule.get("exit_code", 0)) - _log_call(args, rule, exit_code) - return exit_code - - # 2. Unmocked default. - default = manifest.get("unmocked_default") - if isinstance(default, dict): - body = _strip_doc_keys(default.get("response", "")) - exit_code = int(default.get("exit_code", 0)) - sys.stdout.write(body) - _log_call(args, None, exit_code, error="unmocked_default") - return exit_code - - # 3. Legacy error. - _log_call(args, None, 1, error="unmocked") - return _err({"error": "unmocked command", "args": args}, 1) - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/.seal.bin b/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/.seal.bin deleted file mode 100644 index f0f06cb73c..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/.seal.bin +++ /dev/null @@ -1 +0,0 @@ -eNptU11v2yAUffev4A2YWraHqtoiZVKUZmoe1kZx3qoKYfu6YbPBArK1m/bfx4dN6q1+SAwc7j3n3GPZD9o4VAkL11eFTKtvVqvp3R5PTnZ59WKn11+drIrW6B4Nwh39Ao0HO78syvV+uzvwm+0eLeMO4byVHXBOmQGrux9AKBuEAeWK/abc3d+Vm3LEv7r8HmGDi6+ru+2XTXngu9Xh1gPmFzymF0q2YB0L1HFRHu73mwk8r8as0wZwsV6tbze83K/fKsdrUR8z6KY8/FdmQhQNtKgXUhGKLj8jqdyiQP6RLVLaoRlxJm30gNCECY8BdzIKfYgbkwrfLehgnRaNJfMaBkTDHTw7AqrWjVRPS3xy7eVHTGksEjrYBWpk7R6sMxfI/zz6ir//pGNtkHTQe6pz3eyp0xXB75KFryh6KeECU6IHtFz+a/YZGJ5aKyfVCfJmpPOQCwQmKWysur6KEoDE0yisenFgCaWsgXiCha2lxElZ5QkGIZkAXmTLLhCOnfxW/E9iB1F/h+atliG9rNb94MNoSXS7OfWDJaEJZSNqcvYCfRrd9Wbk5IR5NtL4yQvVIBLmnQPD4FlaF6Sc/UmfEuu1z74fCsmFaBwSyZfHXucUs5/GezS6k0QlyFjS9M4AkNk8EyDnyxPnPEyA8zhEzkNqOR/n5z/sQNmRlGX6F1gBP80= \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/.uip.bin b/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/.uip.bin deleted file mode 100644 index d1cc8baf74..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/.uip.bin +++ /dev/null @@ -1 +0,0 @@ -eNrFGWtv4zbyu38FcUAgCaeol6K36BnwAcE2+zikyWKdK9DLGoJs0TY3sqSKdB7N5b/fzJCUSEmOt/1yBhJL5Lw4L86Mxa6uGsWWmeRvfpgI/bbN5LYQS/v6VValfa6kfZLbvRJF+7Zf1k214rLbf2ofldhx+/w7El431Y7VmUIuzGx8gtfJ/O3nj59u0p8+fmYzWgnTdC0KnqZR0nBZFfc8jJI6a3ipJp8v5p+ur+YXcwPvIH/HgiaY/Hx+9fHdxfwm/XR+8wEAfASA2WWlWHOpEjxiMHl7fnmZXl6/t/A+waSoNgjz9sPFKMN0la22PJhcXrw/f/tr6gIOGFvY+c3154sD7KSqGgBJr65TAgOIavmVr1QYTfTK1CzAzlVV8skk52uWbrhKCTeMphMGn01RLbOCaRxaEWvzxoRkZaUIXQPjp+Fq35Q9BATrxE2EJMNYHvhpxWxFfoXiEoQCUFQ8KDbLZYiekeR8Ve1qMLUMtU8myzc/4GLOQ4d7w7M8XT4pLsMoihIDEOzV+vTHIIomnjjPrZ2DKfG97RYWMQvwHBK2nstsx6dswBeeIrauGob7MYNXJkoilICuQ4Mfs+eXKBGK70Cml5fJ8NjaPOBFYP2iCLNmI6dMqiZmzb4AxrkAU/6XbBEz/ihUiuynwEzBe9NUDYEbkBn+i9jpPx3jNSBwk9ORFZ4IAy/Bf2EE50SGsIhfMfq+AhfMU+QNq/ilj0MbQYQ2x0XGC8mNUHDURziRD4/HPwDeHgIQ2ucX61H6SI6PoPC3AS0HCzgFPdF+IUpuvSXf72oZauiodWfH661Tt27Y8TB0OhPzkkxMvtd6HkIlZsf6VMz+4ThaJldCBJq7ap46Bg9CbZmXR5Kq5iVggIMQSVFuZpYoyyRbd8j4WScPDTgRycD+yoIvpWHDH1e8Vux6fuGrrc4g58JRP84v0l/OL/99kb67PH8/Jyc4Pa32qt6r4MU4n6rueCl+56H2PHKfQkh1Cy8LTZNAYLtdBkq3C9rCGGiyB/R+mci6EMoNf3SA7AHSVtYoiXoIg1M8YpmzYBYgEmz3Dltkm5ilMbuH/DQjdMjtSihRgcpm5uT2oyVLshoUmoeIC8waUYfBX74ENuodaYCoz25IBEAO0EA3nr7CXh+1j7kqeFbyfFR78k7UackfMV2/y4B8q1OgjOoxmncV2uL4ooyTsp9VVSpR7rlLyLAY+slhwjfN/ghdc1qrEuARuWnPbBvXw+SQ6qQjQ/o2yc/Jg/CYWv+TXCuQfHRZVcaW5ibSBPr3S6cL2je08EZqHZ822rThgaGnYmIOFaqqk0XbCNdc8CgasG81Zt4J3NCSRg2Q08I6e8IrT2f8mLVZno4K35owFFDgYTkgmJTgZD9DIfIyhFU7kDPMoJBL82qV3vGnsCQuumAY6lRIUULklitOgDGJ5gd3Vj6FoQN3F+sUgmq788I+DfR9SS6H5FxdjenLWSMunuD3mti9JZZA0O71xX9Qegy/oX2O0448TWp30qqsm33JLZ4calM/frM+DYfnu+mAtJEJ1Nsd2ZQV1vu/2QzRyx9U0e0BaVpRFq6Cys7VKBd2aApzVnvFwPd0eFcCsF8BIpJ32YW/oKnpwovZzVOtH4dSI6KbHTwbw/crGKrJBNy1G5AEI4lyJezB9Z8bNWJ86aomCNzDO+HYVxqyjEFjObQps+/x4pdQNaVUNczIqzByLWujQWoJSOI2I/rKM3xNd5bs8r8TZL9SiZItf8zFBsrbMLqdnr1ZtJUnFM3EpcfBKTw1JxAC9LEOnntCRS+mU7I3F/Zw6Blh1+18h9gxG3RBtO5nFDQVUhhrJkYvHM99+qbQPkT0qD1AM4aDksu54Y2PmYoq1lT+Nb+++omqvL6vDSQyzJ3WS2b3vK9hDM8cyjDz4tf1vfq91Vayu8sFXhTY6MoZJkpClSqt7uhVn4P0P2Oulo9YzdwcVB76DYEWM5gaeQ+V77BO5PM0U7bFgJOtqc0ITn49PdmdnuQ3Jx+mJz9PT+b/gbqXYDY73YiYlOSZkoxGl5y22vCm66LJ3nkj5fTxOlmbaQ20UvCRIt1D/WaDzIsAyYt1CiYAPcFW2LXm3RjCWgDyrV8i5xgRFcblvWjgINQlYSsAMkOlaCpnAMBjS14PgyL/o3EAeDQtyZ0xCZu5I4VufVgTD+gPQqPLwz100mr3Arqw1WDuXaV6WpQ8bMUKUiqoHXSBx591aki+VqIMiQbo1ty6QF5tm2q/2boRVTViI0qwH6zdO8W220n7xZR2WawEe1mwrSEMxOhAxJRiEBWmFNPA2rImbsi2bnYx8QI8QQoPo4urmP3NwfFnA/owsRt51BDPAkcr+iS50ykZhbdYxgra2VEBPe+3529BQAP+6cfF+n5UnLJqSQ9louL32TT4U+YiAshvUNQpsMByr0gYhsKgLaqS2ehxMtYLiGBCsKFCohtBJnAbh7dWjgUkDM9hbs+m4CmrrMZRRqo7ZJNiMfvYbDuaXQaegCzNSjRetlsIWDGDqd41ETOHiHnRGkP7GZxRG/RARy1SiHvuNwg9LB1quwyCrxdOfgiBj0DpkSlIh50WwCY4iSnXYoNzoWg0ZhyIwYVsZJYzoFMX2crKOuQGcEe5odb/DDdUKlaALNBZyLqJMSnOlTB0nCGTFVLvHZ442UkjXSQA6s4e9X1VABVvUj1WDjlknGLHRzta9fgzjb5L6ZldF9aWZboTEpqHzfF49sbpzKLFGOhqS4VF44scdUHstNsz7P3DrmWnNB11EyicMlIzbrhRTsVVHMLeLobXqT976E86KXHHpgrqxDhWi5p5p6bkBFww3usOrjLida8jOfJmPkc8iohmD51D6fnzgiRBanolWAwGYjS762f3b83yZvQ74g1HvAIKj7oqoX1CqUa8whW584fxGZylRfEx8sOKS6t/elSnh3+o7fi/agNjxJMy8nXSWd8/i/uLyCsj2WXw5XG9hj8c2etx7mCbI0jfjZUeCiK8+2vL2Zvg1ZnpONaPp1JsDiZjB3FsrBAdLsv87X4J1oXrnyzAjhRZcI1m+wLF9lPTvtxVqztomAxAMDa3MnuDQdGyyp9GFGHATe4znnCsBvWQDulgoFSUIDpyawxr1AOHPqi8cbpnA3p+KdOPLgsF6XoHRsiHNeNZNMGfi1L8LS9NsUcK0hSLnzQNurkryhZSSYRvmKij6H89Mpf4 \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/seal b/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/seal deleted file mode 100644 index d3b053cb06..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/seal +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env python3 -import base64 -import sys -import zlib -from pathlib import Path - -if sys.version_info < (3, 10): - sys.exit(f"seal: requires Python 3.10+ (running {sys.version.split()[0]})") -_bin = Path(__file__).resolve().parent / ".seal.bin" -if not _bin.is_file(): - sys.exit("seal: runtime data missing") -_code = compile(zlib.decompress(base64.b64decode(_bin.read_bytes())), "seal", "exec") -exec(_code, {"__name__": "__main__", "__file__": str(_bin)}) diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/uip b/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/uip deleted file mode 100644 index 6250d33755..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/uip +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env python3 -import base64 -import sys -import zlib -from pathlib import Path - -if sys.version_info < (3, 10): - sys.exit(f"uip: requires Python 3.10+ (running {sys.version.split()[0]})") -_bin = Path(__file__).resolve().parent / ".uip.bin" -if not _bin.is_file(): - sys.exit("uip: runtime data missing") -_code = compile(zlib.decompress(base64.b64decode(_bin.read_bytes())), "uip", "exec") -exec(_code, {"__name__": "__main__", "__file__": str(_bin)}) diff --git a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/uip.cmd b/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/uip.cmd deleted file mode 100644 index 3260cbe41d..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/mock_template/m/uip.cmd +++ /dev/null @@ -1,4 +0,0 @@ -@echo off -REM Windows wrapper so bare `uip` resolves on cmd / PowerShell PATH lookup. -REM Delegates to the python mock script next to this file. -python "%~dp0uip" %* diff --git a/tests/tasks/uipath-troubleshoot/_shared/scripts/compile_mocks.py b/tests/tasks/uipath-troubleshoot/_shared/scripts/compile_mocks.py deleted file mode 100644 index 40a91b2740..0000000000 --- a/tests/tasks/uipath-troubleshoot/_shared/scripts/compile_mocks.py +++ /dev/null @@ -1,87 +0,0 @@ -#!/usr/bin/env python3 -"""Pack the mock scripts in `_shared/mock_src/` into the staged template. - -Each `mock_src/.py` becomes `_shared/mock_template/m/..bin`: -docstrings are stripped from the AST (comments never survive -`ast.unparse`), the stripped source is zlib-compressed and base64-encoded. -The thin loaders at `mock_template/m/uip` and `m/seal` decode and exec the -blob in memory, so the staged sandbox copy documents nothing about the mock -system — a blob shows a single base64 run to `cat`/`strings` (unlike -bytecode, which keeps every string literal readable) and runs on ANY host -CPython >= 3.10 (the sources' syntax floor; the sandbox resolves `python` -from the host PATH, so the artifact must not care which minor runs it). - -Deterministic given the same source and packing interpreter: `ast.unparse` -output is stable within a CPython minor, zlib level 9 and base64 are -deterministic. The canonical committed blobs are produced with CPython -3.13 (the version pinned across the repo's workflows and tests/.venv) so -regenerating unchanged sources is byte-identical and produces no git diff; -the script refuses other minors unless --allow-any-version is passed. - -Run after any edit to `mock_src/*.py`: - - uv run --python 3.13 tests/tasks/uipath-troubleshoot/_shared/scripts/compile_mocks.py -""" - -import argparse -import ast -import base64 -import sys -import zlib -from pathlib import Path - -SHARED_DIR = Path(__file__).resolve().parent.parent -SRC_DIR = SHARED_DIR / "mock_src" -OUT_DIR = SHARED_DIR / "mock_template" / "m" -PACKING_VERSION = (3, 13) - - -def _strip_docstrings(tree: ast.Module) -> ast.Module: - for node in ast.walk(tree): - if not isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): - continue - body = node.body - if ( - body - and isinstance(body[0], ast.Expr) - and isinstance(body[0].value, ast.Constant) - and isinstance(body[0].value.value, str) - ): - body.pop(0) - if not body: - body.append(ast.Pass()) - return tree - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument( - "--allow-any-version", - action="store_true", - help="pack with the running interpreter even if it is not the canonical 3.13", - ) - args = parser.parse_args() - - if sys.version_info[:2] != PACKING_VERSION and not args.allow_any_version: - return ( - f"compile_mocks: pack with CPython {'.'.join(map(str, PACKING_VERSION))} for " - f"byte-stable output (running {sys.version.split()[0]}); e.g. " - "`uv run --python 3.13 ...`, or pass --allow-any-version." - ) - - sources = sorted(SRC_DIR.glob("*.py")) - if not sources: - return f"compile_mocks: no sources found under {SRC_DIR}" - - for src in sources: - tree = _strip_docstrings(ast.parse(src.read_text(encoding="utf-8"))) - stripped = ast.unparse(tree) - blob = base64.b64encode(zlib.compress(stripped.encode("utf-8"), 9)) - out = OUT_DIR / f".{src.stem}.bin" - out.write_bytes(blob) - print(f"packed {src.name} -> {out.relative_to(SHARED_DIR)}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/tasks/uipath-troubleshoot/_shared/scripts/coverage_report.py b/tests/tasks/uipath-troubleshoot/_shared/scripts/coverage_report.py index 0269249944..c979656066 100644 --- a/tests/tasks/uipath-troubleshoot/_shared/scripts/coverage_report.py +++ b/tests/tasks/uipath-troubleshoot/_shared/scripts/coverage_report.py @@ -1,159 +1,171 @@ -"""Coverage reporter: expected vs performed uip commands per replicate. - -Reads: - - //artifacts//m/.log - one record per uip invocation written by the mock dispatcher — - zlib+base64-encoded on sealed runs, plain JSON on unsealed local runs - (legacy artifacts used plain `.calls.jsonl`; both are handled) - - //artifacts//m/r/manifest.json - the manifest's `expected_calls` declares minimum-coverage patterns - (on sealed runs `r/` is gone; the manifest is read from `m/.store`) - -Writes per replicate: - - //coverage.json structured comparison - - //coverage.txt human-readable summary - -Also prints a run-level table to stdout. - -Usage: - python tmp/coverage_report.py runs//default/ - python tmp/coverage_report.py --dump # decoded JSONL +"""Report expected vs performed protected `uip` calls per replicate. + +Reads each replicate's host-side ``protected_mock_calls.jsonl`` and the +scenario's source ``data/uip-fixture.json``. Writes ``coverage.json`` and +``coverage.txt`` beside each replicate and prints a run-level table. """ from __future__ import annotations -import base64 import json +import shlex import sys -import zlib -from collections import Counter, defaultdict +from collections import Counter from pathlib import Path -CALL_LOG_NAMES = (".log", ".calls.jsonl") - -def _parse_call_line(line: str) -> dict | None: - """Parse one call-log line: plain JSON first, then zlib+base64.""" - try: - return json.loads(line) - except json.JSONDecodeError: - pass - try: - return json.loads(zlib.decompress(base64.b64decode(line)).decode("utf-8")) - except (ValueError, zlib.error): - return None +ROOT = Path(__file__).resolve().parents[2] +NOISE_VALUE_FLAGS = frozenset({"--output"}) -def _read_calls(calls_path: Path) -> list[dict]: +def _read_calls(path: Path) -> list[dict]: calls: list[dict] = [] - with calls_path.open(encoding="utf-8") as f: - for line in f: - line = line.strip() - if not line: - continue - rec = _parse_call_line(line) - if rec is not None: - calls.append(rec) + with path.open(encoding="utf-8") as stream: + for line in stream: + if line.strip(): + calls.append(json.loads(line)) return calls -def _find_call_log(sandbox: Path) -> Path | None: - for name in CALL_LOG_NAMES: - path = sandbox / "m" / name - if path.is_file(): - return path - return None +def _match_tokens(argv: list[str]) -> list[str]: + """Mirror the mock service's token expansion: split --flag=value, drop noise flags.""" + expanded: list[str] = [] + for raw in argv: + if raw.startswith("-") and "=" in raw: + flag, value = raw.split("=", 1) + expanded.append(flag) + if value: + expanded.append(value) + else: + expanded.append(raw) + cleaned: list[str] = [] + skip_next = False + for token in expanded: + if skip_next: + skip_next = False + continue + if token in NOISE_VALUE_FLAGS: + skip_next = True + continue + cleaned.append(token) + return cleaned -def _load_manifest(sandbox: Path) -> dict | None: - """Load the manifest from `m/r/` (unsealed) or the sealed `m/.store`.""" - manifest_path = sandbox / "m" / "r" / "manifest.json" - if manifest_path.is_file(): - return json.loads(manifest_path.read_text(encoding="utf-8")) - store_path = sandbox / "m" / ".store" - if store_path.is_file(): - blob = json.loads(zlib.decompress(base64.b64decode(store_path.read_bytes())).decode("utf-8")) - return blob.get("manifest") +def _normalized(argv: list[str]) -> tuple[str, ...]: + return tuple(sorted(_match_tokens(argv))) + + +def _task_fixture(task_id: str) -> Path | None: + marker = f"task_id: {task_id}" + for task_path in ROOT.glob("**/task.yaml"): + if marker not in task_path.read_text(encoding="utf-8"): + continue + fixture = task_path.parent / "data" / "uip-fixture.json" + return fixture if fixture.is_file() else None return None def analyze_replicate(rep_dir: Path) -> dict: """Build a coverage record for one replicate.""" - sandbox = rep_dir / "artifacts" / rep_dir.parent.name - calls_path = _find_call_log(sandbox) - manifest = _load_manifest(sandbox) - + task_id = rep_dir.parent.name + # Host-side per-replicate invocation log written by the protected mock + # client, next to task.json (never inside the agent sandbox). + calls_path = rep_dir / "protected_mock_calls.jsonl" + fixture_path = _task_fixture(task_id) rec: dict = { "replicate": rep_dir.name, - "calls_log_present": calls_path is not None, - "manifest_present": manifest is not None, + "calls_log_present": calls_path.is_file(), + "fixture_present": fixture_path is not None, "expected": [], "calls": [], "missing_expected": [], - "unmocked": [], - "rule_hits": {}, + "unconfigured": [], + "response_hits": {}, "match_rate": None, } - if calls_path is None or manifest is None: + if not calls_path.is_file() or fixture_path is None: return rec - expected = manifest.get("expected_calls", []) + fixture = json.loads(fixture_path.read_text(encoding="utf-8")) calls = _read_calls(calls_path) - - # Per-pattern hit counts expected_status = [] - for spec in expected: - pat = spec.get("pattern", "") + for spec in fixture.get("expected_calls", []): + pattern = spec.get("pattern", "") minimum = spec.get("min", 1) - hits = sum(1 for c in calls if pat in c.get("args", "")) - expected_status.append({ - "pattern": pat, - "min": minimum, - "hits": hits, - "satisfied": hits >= minimum, - "description": spec.get("description", ""), - }) - - # Rule usage - rule_counts: Counter = Counter() - for c in calls: - rule = c.get("matched_rule") - if rule: - rule_counts[rule] += 1 - - # Unmocked calls - unmocked = [c.get("args", "") for c in calls if c.get("matched_rule") is None] + hits = sum(pattern in shlex.join(call.get("argv", [])) for call in calls) + expected_status.append( + { + "pattern": pattern, + "min": minimum, + "hits": hits, + "satisfied": hits >= minimum, + "description": spec.get("description", ""), + } + ) + + exact: dict[tuple[str, ...], str] = {} + normalized: dict[tuple[str, ...], str] = {} + subset_rules: list[tuple[set[str], str]] = [] + for response in fixture.get("responses", []): + argv = response.get("argv", []) + rendered = shlex.join(argv) + mode = response.get("match_mode", "exact") + if mode == "subset": + subset_rules.append((set(_match_tokens(argv)), rendered)) + elif mode == "normalized": + normalized[_normalized(argv)] = rendered + else: + exact[tuple(argv)] = rendered + + response_counts: Counter[str] = Counter() + unconfigured: list[str] = [] + for call in calls: + argv = call.get("argv", []) + rendered = shlex.join(argv) + configured = exact.get(tuple(argv)) or normalized.get(_normalized(argv)) + if not configured and subset_rules: + # Mirror the mock service: subset rules scan in fixture-file order, + # first rule whose tokens all appear in the invocation wins. + invocation_tokens = set(_match_tokens(argv)) + for rule_tokens, rendered_rule in subset_rules: + if rule_tokens <= invocation_tokens: + configured = rendered_rule + break + if configured: + response_counts[configured] += 1 + elif argv[:2] == ["docsai", "ask"]: + response_counts["docsai ask (passthrough)"] += 1 + else: + unconfigured.append(rendered) rec["expected"] = expected_status rec["calls"] = calls - rec["missing_expected"] = [e for e in expected_status if not e["satisfied"]] - rec["unmocked"] = unmocked - rec["rule_hits"] = dict(rule_counts.most_common()) + rec["missing_expected"] = [item for item in expected_status if not item["satisfied"]] + rec["unconfigured"] = unconfigured + rec["response_hits"] = dict(response_counts.most_common()) if expected_status: - rec["match_rate"] = sum(1 for e in expected_status if e["satisfied"]) / len(expected_status) - + rec["match_rate"] = sum(item["satisfied"] for item in expected_status) / len(expected_status) return rec def write_outputs(rep_dir: Path, rec: dict) -> None: (rep_dir / "coverage.json").write_text(json.dumps(rec, indent=2), encoding="utf-8") - lines = [] - lines.append(f"Coverage report for replicate {rec['replicate']}") - lines.append(f" total uip calls : {len(rec['calls'])}") + lines = [ + f"Coverage report for replicate {rec['replicate']}", + f" total uip calls: {len(rec['calls'])}", + ] if rec["match_rate"] is not None: - lines.append(f" expected hit-rate: {rec['match_rate']*100:.0f}% " - f"({len(rec['expected']) - len(rec['missing_expected'])}/{len(rec['expected'])})") + satisfied = len(rec["expected"]) - len(rec["missing_expected"]) + lines.append(f" expected hit-rate: {rec['match_rate'] * 100:.0f}% ({satisfied}/{len(rec['expected'])})") if rec["missing_expected"]: lines.append(" MISSING expected:") - for e in rec["missing_expected"]: - lines.append(f" - {e['pattern']} (got {e['hits']}, need {e['min']})") - if rec["unmocked"]: - lines.append(f" unmocked (exploration): {len(rec['unmocked'])}") - for u in rec["unmocked"][:10]: - lines.append(f" - {u[:100]}") - lines.append(" rule usage:") - for rule, n in rec["rule_hits"].items(): - lines.append(f" {n:>3}x {rule[:80]}") + for item in rec["missing_expected"]: + lines.append(f" - {item['pattern']} (got {item['hits']}, need {item['min']})") + if rec["unconfigured"]: + lines.append(f" unconfigured exploration: {len(rec['unconfigured'])}") + lines.extend(f" - {value[:100]}" for value in rec["unconfigured"][:10]) + lines.append(" response usage:") + lines.extend(f" {count:>3}x {command[:80]}" for command, count in rec["response_hits"].items()) (rep_dir / "coverage.txt").write_text("\n".join(lines) + "\n", encoding="utf-8") @@ -162,49 +174,29 @@ def main(run_dir_arg: str) -> int: if not run_dir.is_dir(): print(f"Run dir not found: {run_dir}", file=sys.stderr) return 2 - - rep_dirs = sorted([d for d in run_dir.iterdir() if d.is_dir() and d.name.isdigit()]) + rep_dirs = sorted(path for path in run_dir.iterdir() if path.is_dir() and path.name.isdigit()) if not rep_dirs: print(f"No replicate directories under {run_dir}", file=sys.stderr) return 2 print(f"Coverage report for {run_dir}\n") - print(f"{'rep':>4} {'calls':>5} {'expected':>14} {'unmocked':>8} {'top exploration':<60}") - print("-" * 100) + print(f"{'rep':>4} {'calls':>5} {'expected':>14} {'unconfigured':>12}") + print("-" * 48) for rep_dir in rep_dirs: rec = analyze_replicate(rep_dir) if not rec["calls_log_present"]: print(f"{rep_dir.name:>4} (no call log)") continue write_outputs(rep_dir, rec) - nm = len(rec["expected"]) - len(rec["missing_expected"]) - tot = len(rec["expected"]) - rate = f"{nm}/{tot} ({(rec['match_rate'] or 0)*100:.0f}%)" - top_unmocked = rec["unmocked"][0] if rec["unmocked"] else "" - print(f"{rep_dir.name:>4} {len(rec['calls']):>5} {rate:>14} {len(rec['unmocked']):>8} {top_unmocked[:60]}") - - return 0 - - -def dump(log_path_arg: str) -> int: - """Print a call log as decoded JSONL (manual triage of sealed runs).""" - log_path = Path(log_path_arg) - if not log_path.is_file(): - print(f"Call log not found: {log_path}", file=sys.stderr) - return 2 - for rec in _read_calls(log_path): - print(json.dumps(rec)) + satisfied = len(rec["expected"]) - len(rec["missing_expected"]) + total = len(rec["expected"]) + rate = f"{satisfied}/{total} ({(rec['match_rate'] or 0) * 100:.0f}%)" + print(f"{rep_dir.name:>4} {len(rec['calls']):>5} {rate:>14} {len(rec['unconfigured']):>12}") return 0 if __name__ == "__main__": - if len(sys.argv) == 3 and sys.argv[1] == "--dump": - sys.exit(dump(sys.argv[2])) if len(sys.argv) != 2: - print( - "Usage: python tmp/coverage_report.py /default/\n" - " python tmp/coverage_report.py --dump ", - file=sys.stderr, - ) + print("Usage: coverage_report.py /default/", file=sys.stderr) sys.exit(2) sys.exit(main(sys.argv[1])) diff --git a/tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py b/tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py index 4d3ad21a4a..ead8eadc9d 100644 --- a/tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py +++ b/tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py @@ -8,17 +8,11 @@ unless `--output` is overridden. NO DIAGNOSIS HINTS IN AGENT-VISIBLE FIXTURES. The sandbox stages `process/` -and `fixtures/` into the agent's cwd, so the agent CAN read `manifest.json`. -Any text you add there - especially the `expected_calls[].description` fields -you fill in after generation - MUST NOT reveal the answer: no exception type, -no fault location (file / activity / DisplayName), no evidence specifics -(e.g. `InputArguments={}`), no hypothesis labels. Keep those descriptions -procedural ("Agent fetches the job's execution details."). The exception and -stack trace belong ONLY in the recorded `jobs get` / `jobs logs` payloads - -that is the evidence the agent must diagnose from. `RESOLUTION.md` lives at the -task root and is NOT staged; `README.md` lives in `data/m/r/`, which IS staged -but is deleted from the sandbox by `m/seal` before the agent starts. Both may -therefore describe the root cause freely. +and `fixtures/` into the agent's cwd. Recorded CLI responses are instead stored +in `data/uip-fixture.json`, which coder-eval mounts only for the protected mock +service. The evaluated agent receives a bounded `uip` client and cannot read +the fixture. `RESOLUTION.md` and `README.md` remain task-authoring files and are +not staged into the agent workspace. Usage: python generate_scenario.py \ @@ -38,15 +32,15 @@ from __future__ import annotations import argparse -import hashlib import json import re -import shutil +import shlex import subprocess import sys from collections import OrderedDict from pathlib import Path + def _find_repo_root() -> Path: """Walk up from this file until we find the repo's `.git` marker.""" p = Path(__file__).resolve().parent @@ -108,28 +102,18 @@ def _find_repo_root() -> Path: allowed_tools: ["Bash", "Read", "Write", "Edit", "Glob", "Grep", "Skill", "Agent", "AskUserQuestion", "TodoWrite"] run_limits: - task_timeout: 2400 + task_timeout: 5400 max_turns: 60 - turn_timeout: 1800 + turn_timeout: 3600 sandbox: - driver: tempdir python: {{}} - template_sources: - - type: template_dir - path: {shared_prefix}_shared/mock_template -{process_source_block} - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true +{template_sources_block} protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md @@ -149,6 +133,7 @@ def _find_repo_root() -> Path: weight: 3.0 pass_threshold: 0.7 include_reference: true + include_dialog: true include_agent_output: true prompt: | Grade the agent's final answer against the attached RESOLUTION.md. @@ -204,10 +189,9 @@ def _find_repo_root() -> Path: | Layer | Source | |---|---| -| `m/uip` + `m/uip.cmd` | shared from the suite `_shared/mock_template/` (manifest-driven dispatcher, shipped as a compressed blob; source in `_shared/mock_src/uip.py`) | +| Protected `uip` mock | coder-eval's host-side protected mock service; the agent receives only a bounded client | | `process/` | frozen snapshot of the failing UiPath project | -| `data/m/r/*.json` | real stdout extracted verbatim from the session transcript (filename = sha1[:10] of args; short to keep Windows paths under MAX_PATH) | -| `data/m/r/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | +| `data/uip-fixture.json` | recorded command map (token-subset matched) with real stdout extracted from the session transcript; private to the mock service | ## Success criteria @@ -397,7 +381,7 @@ def _detect_scrub_map(samples: list[str]) -> "OrderedDict[str, str]": placeholder = "replacement_user" if idx == 1 else f"replacement_user{idx}" user_placeholders[lower] = placeholder for m in WIN_DOMAIN_USER_RE.finditer(text): - domain, user = m.group(1), m.group(2) + _domain, user = m.group(1), m.group(2) # Skip well-known service accounts. if user.upper() in {"SYSTEM", "ADMINISTRATOR", "NETWORKSERVICE", "LOCALSERVICE"}: continue @@ -432,33 +416,54 @@ def _scrub_path(path: Path, mapping: "OrderedDict[str, str]") -> Path: new_parts = [_apply_scrub(p, mapping) for p in parts] return Path(*new_parts) if new_parts else path -def _build_manifest_rules(uip_calls: list[dict]) -> tuple[list[dict], dict[str, str]]: - """One rule per unique `args`. Returns (rules, fixture_filename_by_args). +_NOISE_VALUE_FLAGS = frozenset({"--output"}) + + +def _match_tokens(argv: list[str]) -> list[str]: + """Mirror the mock service's token expansion: split --flag=value, drop noise flags.""" + expanded: list[str] = [] + for raw in argv: + if raw.startswith("-") and "=" in raw: + flag, value = raw.split("=", 1) + expanded.append(flag) + if value: + expanded.append(value) + else: + expanded.append(raw) + cleaned: list[str] = [] + skip_next = False + for token in expanded: + if skip_next: + skip_next = False + continue + if token in _NOISE_VALUE_FLAGS: + skip_next = True + continue + cleaned.append(token) + return cleaned - Fixture filenames are derived from a slug of the args. When two distinct - args slug-collide (rare), a numeric suffix is appended. - """ - rules: list[dict] = [] - fixture_by_args: dict[str, str] = {} - used_filenames: set[str] = set() +def _build_fixture_responses(uip_calls: list[dict]) -> list[dict]: + """Build one subset-matched response per unique recorded argv, most specific first.""" + by_args: "OrderedDict[str, dict]" = OrderedDict() for call in uip_calls: args = call["args"] - if args in fixture_by_args: - continue - slug = hashlib.sha1(args.encode("utf-8")).hexdigest()[:10] - candidate = f"{slug}.json" - n = 2 - while candidate in used_filenames: - candidate = f"{slug}-{n}.json" - n += 1 - used_filenames.add(candidate) - fixture_by_args[args] = candidate - rule = {"match": args, "file": candidate} - if call.get("exit_code") not in (None, 0): - rule["exit_code"] = call["exit_code"] - rules.append(rule) - return rules, fixture_by_args + try: + argv = shlex.split(args, posix=True) + except ValueError as exc: + raise ValueError(f"could not parse recorded uip args {args!r}: {exc}") from exc + by_args[args] = { + "argv": argv, + "match_mode": "subset", + "exit_code": int(call.get("exit_code") or 0), + "stdout": call.get("stdout") or "", + } + responses = list(by_args.values()) + # Subset dispatch is first-match-wins in file order, so the most specific + # rule (largest token set) must come first. Stable sort keeps recording + # order among rules of equal specificity. + responses.sort(key=lambda entry: -len(_match_tokens(entry["argv"]))) + return responses def _snapshot_project( src: Path, dst: Path, mapping: "OrderedDict[str, str] | None" = None @@ -546,13 +551,6 @@ def _build_readme_md(scenario_name: str, summary: str) -> str: summary=summary or "_Add a 1–3 sentence summary of the original investigation here._", ) -def _group_shared_prefix(group: str) -> str: - """`_shared` is at the suite root; a scenario nested `group/scenario/` deep - reaches it with one `../` per path component plus one for the scenario dir.""" - depth = len([p for p in group.split("/") if p]) + 1 if group else 1 - return "../" * depth - - def _group_domain_tags(group: str) -> str: """Default product/domain tag(s) for a group, ready to splice into `tags: [`. Empty when no group (caller adds tags manually).""" @@ -568,14 +566,15 @@ def _group_domain_tags(group: str) -> str: def _build_task_yaml( scenario_name: str, initial_prompt_indented: str, has_project: bool, group: str = "" ) -> str: - process_source_block = ( - " - type: template_dir\n path: process\n" if has_project else "" + template_sources_block = ( + " template_sources:\n - type: template_dir\n path: process\n" + if has_project + else "" ) return TASK_YAML_TEMPLATE.format( slug=scenario_name, initial_prompt_indented=initial_prompt_indented, - process_source_block=process_source_block, - shared_prefix=_group_shared_prefix(group), + template_sources_block=template_sources_block, domain_tags=_group_domain_tags(group), ) @@ -602,14 +601,8 @@ def plan_scenario(args: argparse.Namespace) -> dict: # an empty bash stdout — the real data lives on disk. backfill_stats = _backfill_redirect_stdouts(extracted["uip_calls"], investigation) - # Build manifest + fixtures. - rules, fixture_by_args = _build_manifest_rules(extracted["uip_calls"]) - fixtures: dict[str, str] = {} - for call in extracted["uip_calls"]: - fname = fixture_by_args[call["args"]] - # Last write wins — for duplicate args, we keep the last response, - # but in practice the manifest only references args once. - fixtures[fname] = call["stdout"] + # Build the private recorded command map. + responses = _build_fixture_responses(extracted["uip_calls"]) # Build resolution. resolution_md = _build_resolution_md( @@ -632,8 +625,8 @@ def plan_scenario(args: argparse.Namespace) -> dict: # Aggregate sample text for scrub detection. samples = [resolution_md, readme_md, task_yaml] - samples.extend(fixtures.values()) - samples.extend(rule["match"] for rule in rules) + samples.extend(response["stdout"] for response in responses) + samples.extend(" ".join(response["argv"]) for response in responses) # Snapshot the project (raw — scrub happens after we compute the map). if project is not None: @@ -653,8 +646,14 @@ def plan_scenario(args: argparse.Namespace) -> dict: resolution_md = _apply_scrub(resolution_md, scrub_map) readme_md = _apply_scrub(readme_md, scrub_map) task_yaml = _apply_scrub(task_yaml, scrub_map) - fixtures = {fn: _apply_scrub(c, scrub_map) for fn, c in fixtures.items()} - rules = [{**r, "match": _apply_scrub(r["match"], scrub_map)} for r in rules] + responses = [ + { + **response, + "argv": [_apply_scrub(token, scrub_map) for token in response["argv"]], + "stdout": _apply_scrub(response["stdout"], scrub_map), + } + for response in responses + ] project_plan_scrubbed: list[tuple[Path, str | bytes]] = [] for rel, content in project_plan: rel_scrubbed = _scrub_path(rel, scrub_map) @@ -676,17 +675,11 @@ def plan_scenario(args: argparse.Namespace) -> dict: "extracted_troubleshooting": extracted["troubleshooting"], "backfill_stats": backfill_stats, "scrub_map": scrub_map, - "manifest": { - "version": 2, - "_doc": "Auto-generated by generate_scenario.py - review fixtures + add expected_calls. This file is agent-visible: keep _doc and expected_calls[].description procedural - no exception type, fault location, or root-cause hints (those belong only in the jobs get/logs payloads).", - "rules": rules, - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting.", - }, + "fixture": { + "version": 1, + "responses": responses, + "default": {"stdout": "[]\n", "exit_code": 0}, }, - "fixtures": fixtures, "resolution_md": resolution_md, "readme_md": readme_md, "task_yaml": task_yaml, @@ -715,12 +708,12 @@ def render_dry_run(plan: dict) -> str: for k, v in plan["scrub_map"].items(): out.append(f" {k!r:<60s} -> {v!r}") out.append("") - out.append(f"Manifest rules: {len(plan['manifest']['rules'])}") - for r in plan["manifest"]["rules"][:20]: - suffix = f" exit={r['exit_code']}" if "exit_code" in r else "" - out.append(f" - {r['match']!r:<60s} -> {r['file']}{suffix}") - if len(plan["manifest"]["rules"]) > 20: - out.append(f" ... +{len(plan['manifest']['rules']) - 20} more") + responses = plan["fixture"]["responses"] + out.append(f"Protected fixture responses: {len(responses)}") + for response in responses[:20]: + out.append(f" - {' '.join(response['argv'])!r:<60s} exit={response['exit_code']}") + if len(responses) > 20: + out.append(f" ... +{len(responses) - 20} more") out.append("") out.append(f"Project snapshot: {len(plan['project_files'])} files") out.append("") @@ -728,9 +721,8 @@ def render_dry_run(plan: dict) -> str: base = plan["output_dir"] out.append(f" {base / 'task.yaml'} ({len(plan['task_yaml'])} bytes)") out.append(f" {base / 'RESOLUTION.md'} ({len(plan['resolution_md'])} bytes)") - out.append(f" {base / 'data' / 'm' / 'r' / 'README.md'} ({len(plan['readme_md'])} bytes)") - out.append(f" {base / 'data' / 'm' / 'r' / 'manifest.json'}") - out.append(f" {base / 'data' / 'm' / 'r' / '.json'} x {len(plan['fixtures'])}") + out.append(f" {base / 'README.md'} ({len(plan['readme_md'])} bytes)") + out.append(f" {base / 'data' / 'uip-fixture.json'}") out.append(f" {base / 'process' / ''} x {len(plan['project_files'])}") out.append("") out.append("(dry-run — no files written. Pass --apply to write.)") @@ -747,17 +739,12 @@ def apply_plan(plan: dict) -> None: (base / "task.yaml").write_text(plan["task_yaml"], encoding="utf-8") (base / "RESOLUTION.md").write_text(plan["resolution_md"], encoding="utf-8") - fixtures_dir = base / "data" / "m" / "r" - fixtures_dir.mkdir(parents=True, exist_ok=True) - # README sits beside the fixtures on purpose: `m/seal` deletes the whole - # `r/` dir from the sandbox, so the scenario write-up never reaches the - # agent even though `data/` is staged. - (fixtures_dir / "README.md").write_text(plan["readme_md"], encoding="utf-8") - (fixtures_dir / "manifest.json").write_text( - json.dumps(plan["manifest"], indent=2, ensure_ascii=False) + "\n", encoding="utf-8" + (base / "README.md").write_text(plan["readme_md"], encoding="utf-8") + data_dir = base / "data" + data_dir.mkdir(parents=True, exist_ok=True) + (data_dir / "uip-fixture.json").write_text( + json.dumps(plan["fixture"], indent=2, ensure_ascii=False) + "\n", encoding="utf-8" ) - for fname, content in plan["fixtures"].items(): - (fixtures_dir / fname).write_text(content, encoding="utf-8") process_dir = base / "process" for rel, content in plan["project_files"]: @@ -813,10 +800,8 @@ def main(argv: list[str]) -> int: apply_plan(plan) print(f"Wrote scenario: {plan['output_dir']}") print( - "Next: fill in manifest.json expected_calls, but keep every " - "description procedural - NO exception type / fault location / " - "root-cause hints (manifest.json is agent-visible). The diagnosis " - "lives only in the jobs get/logs payloads." + "Next: review data/uip-fixture.json and RESOLUTION.md. The fixture " + "is mounted only into coder-eval's protected mock service." ) return 0 diff --git a/tests/tasks/uipath-troubleshoot/_shared/scripts/migrate_protected_mocks.py b/tests/tasks/uipath-troubleshoot/_shared/scripts/migrate_protected_mocks.py new file mode 100644 index 0000000000..58ef4a7ed4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/_shared/scripts/migrate_protected_mocks.py @@ -0,0 +1,385 @@ +#!/usr/bin/env python3 +"""Migrate legacy troubleshoot manifests to coder-eval protected mocks. + +The default mode validates every legacy scenario without writing. Pass +``--apply`` to write ``data/uip-fixture.json``, update ``task.yaml``, move the +maintainer README out of the old response tree, and remove ``data/m``. + +This is intentionally a one-way migration. It rejects rules whose token sets +collide with conflicting responses instead of silently choosing a legacy +first-match rule. Emitted rules use ``match_mode: subset`` (rule tokens must +all appear in the invocation) and are sorted most-specific-first because +subset dispatch is first-match-wins in file order. +""" + +from __future__ import annotations + +import argparse +import json +import re +import shlex +import shutil +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +MANIFEST_GLOB = "**/data/m/r/manifest.json" +NOISE_VALUE_FLAGS = frozenset({"--output"}) + + +def _prune_doc_keys(node: object) -> object: + if isinstance(node, dict): + return { + key: _prune_doc_keys(value) + for key, value in node.items() + if not (isinstance(key, str) and key.startswith("_")) + } + if isinstance(node, list): + return [_prune_doc_keys(value) for value in node] + return node + + +def _strip_doc_keys(text: str) -> str: + try: + parsed = json.loads(text) + except (TypeError, ValueError): + return text + pruned = _prune_doc_keys(parsed) + if pruned == parsed: + return text + trailing = "\n" if text.endswith("\n") else "" + return json.dumps(pruned, indent=2, ensure_ascii=False) + trailing + + +def _stdout(value: object) -> str: + if isinstance(value, str): + return _strip_doc_keys(value) + return json.dumps(_prune_doc_keys(value), indent=2, ensure_ascii=False) + "\n" + + +def _argv(match: str) -> list[str]: + try: + argv = shlex.split(match, posix=True) + except ValueError as exc: + raise ValueError(f"invalid command match {match!r}: {exc}") from exc + if not argv: + raise ValueError("command match must not be empty") + return argv + + +def _normalized_key(argv: list[str]) -> tuple[str, ...]: + expanded: list[str] = [] + for raw in argv: + if raw.startswith("-") and "=" in raw: + flag, value = raw.split("=", 1) + expanded.append(flag) + if value: + expanded.append(value) + else: + expanded.append(raw) + + cleaned: list[str] = [] + skip_next = False + for token in expanded: + if skip_next: + skip_next = False + continue + if token in NOISE_VALUE_FLAGS: + skip_next = True + continue + cleaned.append(token) + return tuple(sorted(cleaned)) + + +def _read_response(path: Path) -> str: + raw = path.read_bytes() + if raw.startswith((b"\xff\xfe", b"\xfe\xff")): + return raw.decode("utf-16") + return raw.decode("utf-8-sig", errors="replace") + + +def _build_fixture(manifest_path: Path) -> tuple[dict[str, object], bool]: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + if manifest.get("version") != 2 or not isinstance(manifest.get("rules"), list): + raise ValueError(f"{manifest_path}: expected a version 2 rules manifest") + + responses: list[dict[str, object]] = [] + by_key: dict[tuple[str, ...], dict[str, object]] = {} + docsai_passthrough = False + for index, rule in enumerate(manifest["rules"]): + if not isinstance(rule, dict) or not isinstance(rule.get("match"), str): + raise ValueError(f"{manifest_path}: rule {index} has no string match") + if not rule["match"].strip(): + # Empty legacy matches were unreachable: the dispatcher rejected + # them before either token or substring matching. + continue + argv = _argv(rule["match"]) + if rule.get("passthrough"): + if argv != ["docsai", "ask"]: + raise ValueError(f"{manifest_path}: unsupported passthrough command {argv!r}") + docsai_passthrough = True + continue + filename = rule.get("file") + if not isinstance(filename, str) or not filename: + raise ValueError(f"{manifest_path}: rule {index} has neither file nor passthrough") + response_path = manifest_path.parent / filename + if not response_path.is_file(): + raise ValueError(f"{manifest_path}: missing response file {filename}") + response: dict[str, object] = { + "argv": argv, + "match_mode": "subset", + "exit_code": int(rule.get("exit_code", 0)), + "stdout": _strip_doc_keys(_read_response(response_path)), + } + key = _normalized_key(argv) + previous = by_key.get(key) + if previous is not None: + comparable = {k: v for k, v in response.items() if k != "argv"} + previous_comparable = {k: v for k, v in previous.items() if k != "argv"} + if comparable != previous_comparable: + raise ValueError( + f"{manifest_path}: command token conflict between " + f"{previous['argv']!r} and {argv!r}" + ) + continue + by_key[key] = response + responses.append(response) + + # Subset dispatch is first-match-wins in file order, so the most specific + # rule (largest token set) must come first. Stable sort preserves the + # legacy manifest order among rules of equal specificity. + responses.sort(key=lambda entry: -len(_normalized_key(entry["argv"]))) # type: ignore[arg-type] + + default_raw = manifest.get("unmocked_default") + if isinstance(default_raw, dict): + default = { + "exit_code": int(default_raw.get("exit_code", 0)), + "stdout": _stdout(default_raw.get("response", "")), + } + else: + default = { + "exit_code": 1, + "stderr": '{"error": "unmocked command"}\n', + } + + fixture: dict[str, object] = { + "version": 1, + "responses": responses, + "default": default, + } + expected_calls = manifest.get("expected_calls") + if isinstance(expected_calls, list): + fixture["expected_calls"] = expected_calls + return fixture, docsai_passthrough + + +def _section_bounds(lines: list[str], name: str) -> tuple[int, int]: + start = next((index for index, line in enumerate(lines) if line.rstrip("\r\n") == f"{name}:"), -1) + if start < 0: + raise ValueError(f"task.yaml has no {name} section") + end = len(lines) + top_level = re.compile(r"^[A-Za-z_][A-Za-z0-9_-]*:") + for index in range(start + 1, len(lines)): + if top_level.match(lines[index]): + end = index + break + return start, end + + +def _rewrite_task_yaml(text: str, docsai_passthrough: bool) -> str: + newline = "\r\n" if "\r\n" in text else "\n" + lines = text.splitlines(keepends=True) + sandbox_start, sandbox_end = _section_bounds(lines, "sandbox") + body = lines[sandbox_start + 1 : sandbox_end] + + cleaned: list[str] = [] + index = 0 + while index < len(body): + line = body[index] + stripped = line.strip() + if stripped == "- type: template_dir" and index + 1 < len(body): + path_line = body[index + 1] + path = path_line.strip().removeprefix("path:").strip() + if path == "data" or path.replace("\\", "/").endswith("_shared/mock_template"): + index += 2 + continue + cleaned.extend( + [ + f" - type: template_dir{newline}", + f" path: {path}{newline}", + ] + ) + index += 2 + continue + if stripped.startswith("mock_path_dirs:"): + if cleaned and "Prepend ./m" in cleaned[-1]: + cleaned.pop() + index += 1 + while index < len(body) and body[index].strip() == "- m": + index += 1 + continue + cleaned.append(line) + index += 1 + + for index, line in enumerate(cleaned): + if line.strip() != "template_sources:": + continue + has_source = any(candidate.strip() == "- type: template_dir" for candidate in cleaned[index + 1 :]) + if not has_source: + cleaned.pop(index) + break + + while cleaned and not cleaned[-1].strip(): + cleaned.pop() + protected = [ + f" protected_mocks:{newline}", + f" - tool: uip{newline}", + f" fixture: data/uip-fixture.json{newline}", + f" max_requests: 100{newline}", + ] + if docsai_passthrough: + protected.extend( + [ + f" passthrough_argv_prefixes:{newline}", + f" - [docsai, ask]{newline}", + ] + ) + sandbox = [f"sandbox:{newline}", *cleaned, *protected, newline] + lines[sandbox_start:sandbox_end] = sandbox + + pre_start, pre_end = _section_bounds(lines, "pre_run") + pre_run = "".join(lines[pre_start:pre_end]) + if "python m/seal" not in pre_run: + raise ValueError("task.yaml pre_run does not contain the expected seal command") + if len(re.findall(r"^ - command:", pre_run, flags=re.MULTILINE)) != 1: + raise ValueError("task.yaml pre_run contains commands in addition to seal") + del lines[pre_start:pre_end] + return "".join(lines) + + +def _migrate(manifest_path: Path, *, apply: bool) -> tuple[int, bool]: + fixture, docsai_passthrough = _build_fixture(manifest_path) + task_dir = manifest_path.parents[3] + task_path = task_dir / "task.yaml" + task_text = task_path.read_text(encoding="utf-8") + rewritten = _rewrite_task_yaml(task_text, docsai_passthrough) + if not apply: + return len(fixture["responses"]), docsai_passthrough # type: ignore[arg-type] + + data_dir = task_dir / "data" + data_dir.mkdir(exist_ok=True) + (data_dir / "uip-fixture.json").write_text( + json.dumps(fixture, indent=2, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + task_path.write_text(rewritten, encoding="utf-8") + + old_readme = manifest_path.parent / "README.md" + if old_readme.is_file(): + new_readme = task_dir / "README.md" + if new_readme.exists(): + raise ValueError(f"{task_dir}: both legacy and task-root README exist") + shutil.move(old_readme, new_readme) + shutil.rmtree(task_dir / "data" / "m") + return len(fixture["responses"]), docsai_passthrough # type: ignore[arg-type] + + +def _repair_migrated_tasks(root: Path, *, apply: bool) -> int: + """Remove legacy multiline mock paths and any empty source collection.""" + repaired = 0 + for task_path in root.glob("**/task.yaml"): + text = task_path.read_text(encoding="utf-8") + if "protected_mocks:" not in text: + continue + lines = text.splitlines(keepends=True) + lines = [line for line in lines if line.strip() != "- m"] + orphaned_source = next( + (index for index, line in enumerate(lines) if line.startswith(" - type: template_dir")), + None, + ) + if orphaned_source is not None: + newline = "\r\n" if lines[orphaned_source].endswith("\r\n") else "\n" + lines.insert(orphaned_source, f" template_sources:{newline}") + index = orphaned_source + 1 + while index < len(lines) and not lines[index].startswith(" protected_mocks:"): + if lines[index].startswith(" - type: template_dir"): + lines[index] = " " + lines[index] + elif lines[index].startswith(" path:"): + lines[index] = " " + lines[index] + index += 1 + filtered = "".join(lines) + filtered = re.sub( + r"^ template_sources:\r?\n(?= protected_mocks:)", + "", + filtered, + flags=re.MULTILINE, + ) + if filtered == text: + continue + repaired += 1 + if apply: + task_path.write_text(filtered, encoding="utf-8") + return repaired + + +def _repair_readmes(root: Path, *, apply: bool) -> int: + """Replace obsolete mock-layout descriptions with the protected boundary.""" + markers = ( + "mock_template", + "manifest.json", + "data/m/r", + "mocks/uip", + "m/uip", + "mocks/responses", + "mock dispatcher", + ) + note = ( + "\n## Fixture isolation\n\n" + "`data/uip-fixture.json` is a finite command/response map mounted only into " + "coder-eval's host-side protected mock service. The evaluated agent receives a " + "bounded `uip` client and cannot read the fixture or mock implementation.\n" + ) + repaired = 0 + for readme_path in root.glob("**/README.md"): + if "_shared" in readme_path.parts or not (readme_path.parent / "task.yaml").is_file(): + continue + text = readme_path.read_text(encoding="utf-8") + lines = [line for line in text.splitlines(keepends=True) if not any(marker in line for marker in markers)] + rewritten = "".join(lines).rstrip() + "\n" + if "## Fixture isolation" not in rewritten: + rewritten += note + if rewritten == text: + continue + repaired += 1 + if apply: + readme_path.write_text(rewritten, encoding="utf-8") + return repaired + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=ROOT) + parser.add_argument("--apply", action="store_true") + args = parser.parse_args(argv) + + manifests = sorted(args.root.glob(MANIFEST_GLOB)) + response_count = 0 + passthrough_count = 0 + for manifest in manifests: + responses, passthrough = _migrate(manifest, apply=args.apply) + response_count += responses + passthrough_count += int(passthrough) + repaired = _repair_migrated_tasks(args.root, apply=args.apply) + readmes = _repair_readmes(args.root, apply=args.apply) + action = "Migrated" if args.apply else "Validated" + print( + f"{action} {len(manifests)} scenarios: {response_count} finite responses, " + f"{passthrough_count} docsai passthrough configurations, " + f"{repaired} migrated task YAML repairs, {readmes} README repairs" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/tasks/uipath-troubleshoot/_shared/scripts/verify_manifest_commands.py b/tests/tasks/uipath-troubleshoot/_shared/scripts/verify_manifest_commands.py index ca6ed68da6..d32e41a281 100644 --- a/tests/tasks/uipath-troubleshoot/_shared/scripts/verify_manifest_commands.py +++ b/tests/tasks/uipath-troubleshoot/_shared/scripts/verify_manifest_commands.py @@ -1,13 +1,13 @@ #!/usr/bin/env python3 -"""Verify that every CLI command mocked in a troubleshoot scenario's manifest is +"""Verify that every CLI command in a troubleshoot protected fixture is shaped correctly against the locally installed `uip` CLI. -Command-aggregated: every manifest rule for the same command path collapses +Command-aggregated: every fixture response for the same command path collapses into ONE check. Validity is a property of the command — not of each flag -subset a scenario passed — so all rules for `or jobs list` (`[--folder-key]`, +subset a scenario passed — so all responses for `or jobs list` (`[--folder-key]`, `[--folder-key --output --state]`, …) produce a single `or jobs list [--folder-key --output --state …]` [OK] line that validates the -UNION of every flag seen and the MAX positional count; the affected manifests +UNION of every flag seen and the MAX positional count; the affected fixtures are listed under each command. Three checks per command: @@ -32,7 +32,7 @@ Exits 0 if every check passes; exits 1 otherwise. -Determinism: depends only on (a) the manifests on disk and (b) the tools +Determinism: depends only on (a) the protected fixtures on disk and (b) the tools currently installed in `uip`. No LLM, no network beyond `uip` itself. """ @@ -48,6 +48,7 @@ from pathlib import Path from typing import Iterable + UUID_RE = re.compile(r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$") # Dashless 32-char hex IDs (e.g. trace IDs) are positionals, same as dashed # UUIDs. Without this, an all-[a-f] hex ID that starts with a letter parses @@ -57,7 +58,7 @@ def tokenize_match(match: str) -> tuple[list[str], list[str], list[str]]: - """Split a manifest `match` string into (path, flags, positionals). + """Split a rendered fixture argv into (path, flags, positionals). - path: leading subcommand tokens (plain identifiers before any flag / UUID / quoted arg). @@ -129,7 +130,7 @@ def fetch_help(uip_bin: str, prefix: tuple[str, ...]) -> dict | None: # Aspect routers that JSON --help hides — the parent's JSON payload doesn't # list these in Subcommands, yet they're real subcommands at runtime. Caught -# by the user via the maestro-stuck-rpa-job manifest which calls real +# by the user via the maestro-stuck-rpa-job fixture which calls real # `uip maestro bpmn instance ...` paths. Add more entries here if other tools # exhibit the same routing quirk. KNOWN_ASPECT_ROUTERS: dict[tuple[str, ...], set[str]] = { @@ -311,7 +312,7 @@ def validate_shape( if expected_args and total_positionals > expected_args: return False, ( f"'uip {' '.join(effective_path)}' expects {expected_args} positional arg(s); " - f"manifest supplies {total_positionals}" + f"fixture supplies {total_positionals}" ) return True, "" @@ -329,22 +330,39 @@ def shape_repr(path: tuple[str, ...], flags: frozenset[str], positional_count: i return " ".join(parts) -def walk_manifests(root: Path) -> Iterable[dict]: - """Yield one entry per rule.match across every troubleshoot manifest.""" - for manifest_path in sorted(root.glob("**/data/m/r/manifest.json")): - scenario = manifest_path.parents[3].name +def walk_fixtures(root: Path) -> Iterable[dict]: + """Yield one entry per argv response and configured docsai passthrough.""" + for fixture_path in sorted(root.glob("**/data/uip-fixture.json")): + scenario = fixture_path.parents[1].name try: - manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + fixture = json.loads(fixture_path.read_text(encoding="utf-8")) except json.JSONDecodeError as exc: - yield {"scenario": scenario, "manifest": manifest_path, "parse_error": str(exc)} + yield {"scenario": scenario, "fixture": fixture_path, "parse_error": str(exc)} continue - for idx, rule in enumerate(manifest.get("rules", [])): + for idx, response in enumerate(fixture.get("responses", [])): + argv = response.get("argv", []) + if not isinstance(argv, list) or not all(isinstance(token, str) for token in argv): + yield { + "scenario": scenario, + "fixture": fixture_path, + "parse_error": f"response {idx} has invalid argv", + } + continue yield { "scenario": scenario, - "manifest": manifest_path, + "fixture": fixture_path, "rule_index": idx, - "match": rule.get("match", ""), - "passthrough": bool(rule.get("passthrough")), + "match": shlex.join(argv), + "passthrough": False, + } + task_text = (fixture_path.parents[1] / "task.yaml").read_text(encoding="utf-8") + if "- [docsai, ask]" in task_text: + yield { + "scenario": scenario, + "fixture": fixture_path, + "rule_index": "passthrough", + "match": "docsai ask", + "passthrough": True, } @@ -363,18 +381,18 @@ def main(argv: list[str] | None = None) -> int: print(f"error: root {args.root} does not exist", file=sys.stderr) return 2 - # Aggregate manifest rules by COMMAND PATH. Whether a command is valid is + # Aggregate fixture responses by COMMAND PATH. Whether a command is valid is # a property of the command itself — does the leaf exist, does it accept # each flag, does it accept the positional arity — NOT of each individual # flag-subset a scenario happened to pass. So every rule for a command # collapses into ONE check that validates the UNION of all flags seen and - # the MAX positional count across its rules. `or jobs list [--folder-key]`, + # the MAX positional count across its responses. `or jobs list [--folder-key]`, # `[--folder-key --output --state]`, etc. become a single # `or jobs list [--folder-key --output --state ...]` check. commands: dict[tuple[str, ...], dict] = {} parse_errors: list[dict] = [] total_rules = 0 - for entry in walk_manifests(args.root): + for entry in walk_fixtures(args.root): if "parse_error" in entry: parse_errors.append(entry) continue @@ -390,9 +408,9 @@ def main(argv: list[str] | None = None) -> int: agg["entries"].append(entry) command_keys = sorted(commands.keys()) - print(f"Manifests scanned under: {args.root}") - print(f"Manifest parse errors: {len(parse_errors)}") - print(f"Rules across manifests: {total_rules}") + print(f"Fixtures scanned under: {args.root}") + print(f"Fixture parse errors: {len(parse_errors)}") + print(f"Responses across fixtures: {total_rules}") print(f"Distinct commands: {len(command_keys)}") print() @@ -429,13 +447,13 @@ def main(argv: list[str] | None = None) -> int: if parse_errors: print() - print("Manifest parse errors:") + print("Fixture parse errors:") for e in parse_errors: - print(f" [{e['scenario']}] {e['manifest']}: {e['parse_error']}") + print(f" [{e['scenario']}] {e['fixture']}: {e['parse_error']}") if bad_commands: print() - print("Invalid commands — affected manifest rules:") + print("Invalid commands — affected fixture responses:") for path, flags, positional_count, msg in bad_commands: print(f" {shape_repr(path, flags, positional_count)} ({msg})") for entry in commands[path]["entries"]: diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/README.md new file mode 100644 index 0000000000..fa70ee5b57 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/README.md @@ -0,0 +1,32 @@ +# classic-browserscope-efail + +Faithful-replay scenario for the `uipath-troubleshoot` skill. + +## What this scenario covers + +A scheduled **unattended** run of `PortalReconciliation` (folder `Finance`) faults ~7 s in at a classic +`Attach Browser` scope (`UiPath.Core.Activities.BrowserScope`) with a +`System.Runtime.InteropServices.COMException` — "Invalid access to memory location." +(HRESULT `0x800703E6`), wrapped as `BrowserOperationException`. The prior run of the same process +succeeded as an **attended** run on a different machine. + +The correct diagnosis is an **environmental COM conflict** at the classic BrowserScope (display-scaling +/ privilege-level / non-interactive-session mismatch between the dev machine and the production robot), +**not** a selector, timeout, or extension defect — the source shows valid selectors and a well-formed +scope. Immediate remediation is environment alignment (recommendation only); strategic remediation is +migrating to the modern `Use Application/Browser` container. + +Exercises the `activity-packages/classic-activities/playbooks/browser-scope-errors.md` playbook, which +owns the COM/environmental BrowserScope family and its cross-references to the extension/comms +(`browser-open-or-attach-failed.md`) and selector/timeout (`ui-element-not-found.md`, +`ui-activity-timeout.md`) siblings. + +## Evidence layout + +- `process/` — the failing project source (classic `BrowserScope` + valid child `Get Text`) + +Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/3db6b96527.json deleted file mode 100644 index e3fef7de17..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/3db6b96527.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7c4e0b9a-3f21-4d88-a6b2-19e5c3d40f77", - "StartTime": "2026-07-15T02:15:04.180Z", - "EndTime": "2026-07-15T02:15:11.902Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "7c4e0b9a-3f21-4d88-a6b2-19e5c3d40f77", - "Info": "Attach Browser 'Chrome portal': Could not attach to the browser. Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\r\n\r\nin Main.xaml\r\n at Attach Browser \"Attach Browser 'Chrome portal'\"\r\n at Sequence \"Reconcile\"\r\n at DynamicActivity \"Main\"\r\n\r\nUiPath.Core.Activities.BrowserOperationException: Could not attach to the browser. Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\r\n ---> System.Runtime.InteropServices.COMException: Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\r\n at UiPath.UIAutomationCore.UiNode.SetForeground()\r\n at UiPath.Core.UiElement.SetForeground()\r\n at UiPath.Core.Activities.BrowserScope.AttachToBrowser(Browser browser, NativeActivityContext context)\r\n at UiPath.Core.Activities.BrowserScope.OnBrowserFound(NativeActivityContext context, Browser browser)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Core.Activities.BrowserScope.OnBrowserFound(NativeActivityContext context, Browser browser)\r\n at UiPath.Core.Activities.BrowserScope.Execute(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-07-15T02:15:03.900Z", - "ReleaseName": "PortalReconciliation", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21093344, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "a2f9c1d4-6b08-4e17-9c3a-2d5e8f0b7143", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-15T02:15:11.902Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7c4e0b9a3f214d88a6b219e5c3d40f77", - "Id": 66214788 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/701433db1c.json deleted file mode 100644 index ff3626daec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/701433db1c.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-15T02:15:11.930Z", - "JobId": 66214788 - }, - { - "State": "Running", - "Timestamp": "2026-07-15T02:15:04.470Z", - "JobId": 66214788 - }, - { - "State": "Pending", - "Timestamp": "2026-07-15T02:15:03.900Z", - "JobId": 66214788 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/75949d659e.json deleted file mode 100644 index e5658e8556..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/75949d659e.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7c4e0b9a-3f21-4d88-a6b2-19e5c3d40f77", - "State": "Faulted", - "ProcessName": "PortalReconciliation", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-15T02:15:04.180Z", - "EndTime": "2026-07-15T02:15:11.902Z", - "CreationTime": "2026-07-15T02:15:03.900Z" - }, - { - "Key": "1b90d6f2-8c44-4a1e-b7d3-6f0a2e9c5511", - "State": "Successful", - "ProcessName": "PortalReconciliation", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Attended", - "Priority": "Normal", - "Machine": "DEV-BOX", - "MachineKey": "", - "StartTime": "2026-07-14T16:42:22.501Z", - "EndTime": "2026-07-14T16:43:07.815Z", - "CreationTime": "2026-07-14T16:42:22.501Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/README.md deleted file mode 100644 index d8fcb53bbb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# classic-browserscope-efail - -Faithful-replay scenario for the `uipath-troubleshoot` skill. - -## What this scenario covers - -A scheduled **unattended** run of `PortalReconciliation` (folder `Finance`) faults ~7 s in at a classic -`Attach Browser` scope (`UiPath.Core.Activities.BrowserScope`) with a -`System.Runtime.InteropServices.COMException` — "Invalid access to memory location." -(HRESULT `0x800703E6`), wrapped as `BrowserOperationException`. The prior run of the same process -succeeded as an **attended** run on a different machine. - -The correct diagnosis is an **environmental COM conflict** at the classic BrowserScope (display-scaling -/ privilege-level / non-interactive-session mismatch between the dev machine and the production robot), -**not** a selector, timeout, or extension defect — the source shows valid selectors and a well-formed -scope. Immediate remediation is environment alignment (recommendation only); strategic remediation is -migrating to the modern `Use Application/Browser` container. - -Exercises the `activity-packages/classic-activities/playbooks/browser-scope-errors.md` playbook, which -owns the COM/environmental BrowserScope family and its cross-references to the extension/comms -(`browser-open-or-attach-failed.md`) and selector/timeout (`ui-element-not-found.md`, -`ui-activity-timeout.md`) siblings. - -## Evidence layout - -- `data/m/r/folders.json` — `or folders list` (Finance folder key) -- `data/m/r/joblist.json` — `or jobs list` (faulted unattended job + prior successful attended job) -- `data/m/r/jobget.json` — `or jobs get` (COMException `0x800703E6` stack + Unattended/MOCK-HOST metadata) -- `data/m/r/joblogs.json` — `or jobs logs` (Error log with the COM stack) -- `data/m/r/jobhist.json` — `or jobs history` (Pending → Running → Faulted) -- `process/` — the failing project source (classic `BrowserScope` + valid child `Get Text`) - -Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/b8d18266dc.json deleted file mode 100644 index 335d501682..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-15T02:15:11.640Z", - "Message": "Attach Browser 'Chrome portal': Could not attach to the browser. Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\r\nin Main.xaml\r\n at Attach Browser \"Attach Browser 'Chrome portal'\"\r\n at Sequence \"Reconcile\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.Core.Activities.BrowserOperationException ---> System.Runtime.InteropServices.COMException (0x800703E6): Invalid access to memory location.\r\n at UiPath.UIAutomationCore.UiNode.SetForeground()\r\n at UiPath.Core.Activities.BrowserScope.AttachToBrowser(Browser browser, NativeActivityContext context)\r\n at UiPath.Core.Activities.BrowserScope.Execute(NativeActivityContext context)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-07-15T02:15:05.010Z", - "Message": "Attach Browser 'Chrome portal' execution started" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-15T02:15:04.520Z", - "Message": "PortalReconciliation execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/ee2628e6b6.json deleted file mode 100644 index 35caf38839..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a2f9c1d4-6b08-4e17-9c3a-2d5e8f0b7143", - "Name": "Finance", - "Path": "Finance", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/manifest.json deleted file mode 100644 index b79fa5748c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored fixtures for a faithful-replay of a classic BrowserScope (Attach Browser) COM/environmental fault (job 7c4e0b9a, process PortalReconciliation, folder Finance). The faulted unattended run (MOCK-HOST) plus a prior successful attended run (DEV-BOX) supply the environment delta. Matching is token-aware with plain-substring fallback (first match wins).", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/uip-fixture.json new file mode 100644 index 0000000000..f773ce7a2c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a2f9c1d4-6b08-4e17-9c3a-2d5e8f0b7143\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7c4e0b9a-3f21-4d88-a6b2-19e5c3d40f77\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PortalReconciliation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-15T02:15:04.180Z\",\r\n \"EndTime\": \"2026-07-15T02:15:11.902Z\",\r\n \"CreationTime\": \"2026-07-15T02:15:03.900Z\"\r\n },\r\n {\r\n \"Key\": \"1b90d6f2-8c44-4a1e-b7d3-6f0a2e9c5511\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"PortalReconciliation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DEV-BOX\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-14T16:42:22.501Z\",\r\n \"EndTime\": \"2026-07-14T16:43:07.815Z\",\r\n \"CreationTime\": \"2026-07-14T16:42:22.501Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-15T02:15:11.640Z\",\r\n \"Message\": \"Attach Browser 'Chrome portal': Could not attach to the browser. Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\\r\\nin Main.xaml\\r\\n at Attach Browser \\\"Attach Browser 'Chrome portal'\\\"\\r\\n at Sequence \\\"Reconcile\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.Core.Activities.BrowserOperationException ---> System.Runtime.InteropServices.COMException (0x800703E6): Invalid access to memory location.\\r\\n at UiPath.UIAutomationCore.UiNode.SetForeground()\\r\\n at UiPath.Core.Activities.BrowserScope.AttachToBrowser(Browser browser, NativeActivityContext context)\\r\\n at UiPath.Core.Activities.BrowserScope.Execute(NativeActivityContext context)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-07-15T02:15:05.010Z\",\r\n \"Message\": \"Attach Browser 'Chrome portal' execution started\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-15T02:15:04.520Z\",\r\n \"Message\": \"PortalReconciliation execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-15T02:15:11.930Z\",\r\n \"JobId\": 66214788\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-15T02:15:04.470Z\",\r\n \"JobId\": 66214788\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-15T02:15:03.900Z\",\r\n \"JobId\": 66214788\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7c4e0b9a-3f21-4d88-a6b2-19e5c3d40f77\",\r\n \"StartTime\": \"2026-07-15T02:15:04.180Z\",\r\n \"EndTime\": \"2026-07-15T02:15:11.902Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"7c4e0b9a-3f21-4d88-a6b2-19e5c3d40f77\",\r\n \"Info\": \"Attach Browser 'Chrome portal': Could not attach to the browser. Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\\r\\n\\r\\nin Main.xaml\\r\\n at Attach Browser \\\"Attach Browser 'Chrome portal'\\\"\\r\\n at Sequence \\\"Reconcile\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.BrowserOperationException: Could not attach to the browser. Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\\r\\n ---> System.Runtime.InteropServices.COMException: Invalid access to memory location. (Exception from HRESULT: 0x800703E6)\\r\\n at UiPath.UIAutomationCore.UiNode.SetForeground()\\r\\n at UiPath.Core.UiElement.SetForeground()\\r\\n at UiPath.Core.Activities.BrowserScope.AttachToBrowser(Browser browser, NativeActivityContext context)\\r\\n at UiPath.Core.Activities.BrowserScope.OnBrowserFound(NativeActivityContext context, Browser browser)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Core.Activities.BrowserScope.OnBrowserFound(NativeActivityContext context, Browser browser)\\r\\n at UiPath.Core.Activities.BrowserScope.Execute(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-07-15T02:15:03.900Z\",\r\n \"ReleaseName\": \"PortalReconciliation\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21093344,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"a2f9c1d4-6b08-4e17-9c3a-2d5e8f0b7143\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-15T02:15:11.902Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7c4e0b9a3f214d88a6b219e5c3d40f77\",\r\n \"Id\": 66214788\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/task.yaml index d9bb3a2850..86101f40c6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-browserscope-efail/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/README.md new file mode 100644 index 0000000000..5de77c8e9d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/README.md @@ -0,0 +1,52 @@ +# Classic Click No-Op (SimulateClick on Java) — Synthetic Scenario + +A classic `Click` (`UiPath.Core.Activities.Click`) with `SimulateClick=True` +targets a **Java** desktop app, where SimulateClick is unsupported — so the +click reports `Successful` while the Submit button is never actuated: job +`Successful`, zero Error logs, claim status still `Draft`. Classic Click has no +Verify Execution, so the miss is silent by design. + +Exercises the +`skills/uipath-troubleshoot/references/activity-packages/classic-activities/playbooks/click-silent-no-op.md` +playbook and the no-signature routing path. + +## What the agent must uncover + +**Root Cause:** `SimulateClick=True` on a Java target is a documented silent +no-op; classic Click has no Verify Execution to catch it. **Fix:** disable +SimulateClick (use Default / hardware events) and add an explicit post-click +check. See `RESOLUTION.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing project (classic `Click`, `SimulateClick=True`, Java selector) | + +Synthetic (not a faithful replay): fixtures are hand-authored so the root cause +is confirmable from runtime evidence (status still Draft) plus the workflow +source (SimulateClick on a Java target). The agent-visible surface +(`process/` + `data/`) carries no diagnosis hints — the input-method/target-tech +knowledge lives in the skill's playbook, not in a planted fixture. + +## Success criteria + +Scores the **conclusion**, not the trajectory (suite two-criteria convention): + +- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). +- Agent reached the same root cause + fix as `RESOLUTION.md` (`llm_judge`, threshold 0.7). + +## Running + +```bash +SKILLS_REPO_PATH="C:\\Work\\UiPath" .venv/Scripts/coder-eval.exe run \ + tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/task.yaml \ + -e experiments/default.yaml --repeats 3 -j 3 -v +``` + +Requires `SKILLS_REPO_PATH` and a tilde-free `TMPDIR/TEMP/TMP` (e.g. `C:/cetmp`) +on Windows. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/10fold.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/10fold.json deleted file mode 100644 index 27085deb3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/10fold.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f1a2b3c4-d5e6-4f70-8a1b-2c3d4e5f6a7b", - "DisplayName": "Finance", - "FullyQualifiedName": "Finance", - "ProvisionType": "Manual", - "PermissionModel": "InheritFromTenant", - "Id": 771204 - }, - { - "Key": "c0ffee11-2233-4455-6677-8899aabbccdd", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "ProvisionType": "Automatic", - "PermissionModel": "InheritFromTenant", - "Id": 1500287 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/20jobs.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/20jobs.json deleted file mode 100644 index 4901ef7997..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/20jobs.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e", - "State": "Successful", - "ProcessName": "ClassicClaimsSubmit", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-15T21:04:12.480Z", - "EndTime": "2026-07-15T21:04:49.905Z", - "CreationTime": "2026-07-15T21:04:11.900Z" - }, - { - "Key": "8e1d55a2-3c47-4a90-b2f1-60a9c8d7e6f5", - "State": "Successful", - "ProcessName": "ClassicClaimsSubmit", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-14T21:03:58.100Z", - "EndTime": "2026-07-14T21:04:35.640Z", - "CreationTime": "2026-07-14T21:03:57.500Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/30jobg.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/30jobg.json deleted file mode 100644 index 59cc4dc3f4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/30jobg.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e", - "StartTime": "2026-07-15T21:04:12.480Z", - "EndTime": "2026-07-15T21:04:49.905Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e", - "Info": "Job completed", - "CreationTime": "2026-07-15T21:04:11.900Z", - "ReleaseName": "ClassicClaimsSubmit", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 41220876, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 771204, - "FolderKey": "f1a2b3c4-d5e6-4f70-8a1b-2c3d4e5f6a7b", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svc_rpa", - "OrchestratorUserIdentity": "rpa-service@contoso.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-15T21:04:49.905Z", - "Id": 90551238 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/40errl.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/40errl.json deleted file mode 100644 index 26238d5ebb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/40errl.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/50infl.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/50infl.json deleted file mode 100644 index fe4f5042dd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/50infl.json +++ /dev/null @@ -1,57 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "Message": "ClassicClaimsSubmit execution started", - "TimeStamp": "2026-07-15T21:04:12.600Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "ClassicClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Click 'Submit Claim' execution started", - "TimeStamp": "2026-07-15T21:04:15.010Z", - "RobotName": "svc_rpa", - "ProcessName": "ClassicClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Click 'Submit Claim' execution ended", - "TimeStamp": "2026-07-15T21:04:15.470Z", - "RobotName": "svc_rpa", - "ProcessName": "ClassicClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Get Text 'Claim Status' execution ended", - "TimeStamp": "2026-07-15T21:04:16.220Z", - "RobotName": "svc_rpa", - "ProcessName": "ClassicClaimsSubmit" - }, - { - "Level": "Information", - "Message": "Claim status after submit: Draft", - "TimeStamp": "2026-07-15T21:04:16.300Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "ClassicClaimsSubmit" - }, - { - "Level": "Information", - "Message": "ClassicClaimsSubmit execution ended", - "TimeStamp": "2026-07-15T21:04:49.800Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "ClassicClaimsSubmit" - } - ], - "Pagination": { - "Returned": 6, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/README.md deleted file mode 100644 index 88c831f277..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Classic Click No-Op (SimulateClick on Java) — Synthetic Scenario - -A classic `Click` (`UiPath.Core.Activities.Click`) with `SimulateClick=True` -targets a **Java** desktop app, where SimulateClick is unsupported — so the -click reports `Successful` while the Submit button is never actuated: job -`Successful`, zero Error logs, claim status still `Draft`. Classic Click has no -Verify Execution, so the miss is silent by design. - -Exercises the -`skills/uipath-troubleshoot/references/activity-packages/classic-activities/playbooks/click-silent-no-op.md` -playbook and the no-signature routing path. - -## What the agent must uncover - -**Root Cause:** `SimulateClick=True` on a Java target is a documented silent -no-op; classic Click has no Verify Execution to catch it. **Fix:** disable -SimulateClick (use Default / hardware events) and add an explicit post-click -check. See `RESOLUTION.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing project (classic `Click`, `SimulateClick=True`, Java selector) | -| `data/m/r/*.json` | canned `uip` CLI stdout: folders, jobs list/get, Error logs (empty), Info logs (status still Draft) | -| `data/m/r/manifest.json` | dispatch table; `docsai ask` proxies to the real CLI (passthrough) | - -Synthetic (not a faithful replay): fixtures are hand-authored so the root cause -is confirmable from runtime evidence (status still Draft) plus the workflow -source (SimulateClick on a Java target). The agent-visible surface -(`process/` + `data/`) carries no diagnosis hints — the input-method/target-tech -knowledge lives in the skill's playbook, not in a planted fixture. - -## Success criteria - -Scores the **conclusion**, not the trajectory (suite two-criteria convention): - -- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). -- Agent reached the same root cause + fix as `RESOLUTION.md` (`llm_judge`, threshold 0.7). - -## Running - -```bash -SKILLS_REPO_PATH="C:\\Work\\UiPath" .venv/Scripts/coder-eval.exe run \ - tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/task.yaml \ - -e experiments/default.yaml --repeats 3 -j 3 -v -``` - -Requires `SKILLS_REPO_PATH` and a tilde-free `TMPDIR/TEMP/TMP` (e.g. `C:/cetmp`) -on Windows. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/manifest.json deleted file mode 100644 index 66e93f8d82..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/m/r/manifest.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "version": 2, - "rules": [ - { - "match": "or folders list", - "file": "10fold.json" - }, - { - "match": "or jobs list", - "file": "20jobs.json" - }, - { - "match": "or jobs get 3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e", - "file": "30jobg.json" - }, - { - "match": "or jobs logs 3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e --level Error", - "file": "40errl.json" - }, - { - "match": "or jobs logs 3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e", - "file": "50infl.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/uip-fixture.json new file mode 100644 index 0000000000..5de3124467 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/data/uip-fixture.json @@ -0,0 +1,64 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e\",\r\n \"StartTime\": \"2026-07-15T21:04:12.480Z\",\r\n \"EndTime\": \"2026-07-15T21:04:49.905Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-07-15T21:04:11.900Z\",\r\n \"ReleaseName\": \"ClassicClaimsSubmit\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 41220876,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 771204,\r\n \"FolderKey\": \"f1a2b3c4-d5e6-4f70-8a1b-2c3d4e5f6a7b\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_rpa\",\r\n \"OrchestratorUserIdentity\": \"rpa-service@contoso.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-15T21:04:49.905Z\",\r\n \"Id\": 90551238\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"ClassicClaimsSubmit execution started\",\r\n \"TimeStamp\": \"2026-07-15T21:04:12.600Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Click 'Submit Claim' execution started\",\r\n \"TimeStamp\": \"2026-07-15T21:04:15.010Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Click 'Submit Claim' execution ended\",\r\n \"TimeStamp\": \"2026-07-15T21:04:15.470Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Get Text 'Claim Status' execution ended\",\r\n \"TimeStamp\": \"2026-07-15T21:04:16.220Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"Claim status after submit: Draft\",\r\n \"TimeStamp\": \"2026-07-15T21:04:16.300Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"ClassicClaimsSubmit execution ended\",\r\n \"TimeStamp\": \"2026-07-15T21:04:49.800Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 6,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f1a2b3c4-d5e6-4f70-8a1b-2c3d4e5f6a7b\",\r\n \"DisplayName\": \"Finance\",\r\n \"FullyQualifiedName\": \"Finance\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"InheritFromTenant\",\r\n \"Id\": 771204\r\n },\r\n {\r\n \"Key\": \"c0ffee11-2233-4455-6677-8899aabbccdd\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"ProvisionType\": \"Automatic\",\r\n \"PermissionModel\": \"InheritFromTenant\",\r\n \"Id\": 1500287\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3b9c2d1e-4f5a-4b6c-8d7e-9f0a1b2c3d4e\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-15T21:04:12.480Z\",\r\n \"EndTime\": \"2026-07-15T21:04:49.905Z\",\r\n \"CreationTime\": \"2026-07-15T21:04:11.900Z\"\r\n },\r\n {\r\n \"Key\": \"8e1d55a2-3c47-4a90-b2f1-60a9c8d7e6f5\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ClassicClaimsSubmit\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-14T21:03:58.100Z\",\r\n \"EndTime\": \"2026-07-14T21:04:35.640Z\",\r\n \"CreationTime\": \"2026-07-14T21:03:57.500Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/task.yaml index 8078209867..48b565bf7b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-click-noop-simulate/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/README.md new file mode 100644 index 0000000000..4e72c353eb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/README.md @@ -0,0 +1,36 @@ +# Classic Invoke Workflow File — Runtime Argument Mismatch (Stale Key) + +Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. + +## What this scenario exercises + +A job faults at an `Invoke Workflow File` with `System.ArgumentException` — "the following keys from the +input dictionary do not map to arguments and must be removed: in_Amount". The agent must recognize +**argument-name drift**: the parent passes `in_Amount`, but the invoked `ValidatePayment.xaml` renamed +that input to `in_GrossAmount`. The invoke fails at argument binding, before the child body runs. Fix: +Import Arguments and remap to `in_GrossAmount`. + +Distinct from the design-time `classic-invoke-arg-required` scenario (a Studio validation error about an +*unmapped required* argument): here the parent maps a *stale key that no longer exists* on the child, +and the failure is a **runtime** `ArgumentException`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` invoking `ValidatePayment.xaml` with `in_Amount`; the child declares `in_GrossAmount` | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the stale-argument-key mismatch and the fix (Import Arguments, remap to + `in_GrossAmount`). + +Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` (runtime +"argument name/type/direction mismatch" branch). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/244ab94cfe.json deleted file mode 100644 index ef0628fea7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-16T14:41:09.010Z", - "JobId": 66123980 - }, - { - "State": "Running", - "Timestamp": "2026-07-16T14:41:08.470Z", - "JobId": 66123980 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/3672254a58.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/3672254a58.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/3672254a58.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/464db476bb.json deleted file mode 100644 index fae833b519..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/464db476bb.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4e9d2a71-6c05-4f38-9b12-8a3f0e7c1d59", - "StartTime": "2026-07-16T14:41:08.220Z", - "EndTime": "2026-07-16T14:41:09.010Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "Info": "Invoke ValidatePayment: The values provided for the root activity's arguments did not satisfy the root activity's requirements: The following keys from the input dictionary do not map to arguments and must be removed: in_Amount. Please note that argument names are case sensitive.\r\nParameter name: rootArgumentValues\r\n\r\nin Main.xaml\r\n at InvokeWorkflowFile \"Invoke ValidatePayment\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\nSystem.ArgumentException: The values provided for the root activity's arguments did not satisfy the root activity's requirements: The following keys from the input dictionary do not map to arguments and must be removed: in_Amount. Please note that argument names are case sensitive.\r\nParameter name: rootArgumentValues\r\n at System.Activities.Hosting.WorkflowInstance.RegisterExtensionManager(WorkflowInstanceExtensionManager extensionManager)\r\n at System.Activities.WorkflowApplication.EnsureInitialized()\r\n at UiPath.Core.Activities.InvokeWorkflowFile.d__.MoveNext()", - "CreationTime": "2026-07-16T14:41:08.220Z", - "ReleaseName": "PaymentValidation", - "Type": "Attended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-16T14:41:09.010Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "4e9d2a716c054f389b128a3f0e7c1d59", - "Id": 66123980 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/6059da6ae1.json deleted file mode 100644 index c02eebc498..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4e9d2a71-6c05-4f38-9b12-8a3f0e7c1d59", - "State": "Faulted", - "ProcessName": "PaymentValidation", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-16T14:41:08.220Z", - "EndTime": "2026-07-16T14:41:09.010Z", - "CreationTime": "2026-07-16T14:41:08.220Z" - }, - { - "Key": "9b0e4c22-1a76-4d38-8f05-2c7a1e9b6d43", - "State": "Successful", - "ProcessName": "PaymentValidation", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-10T11:20:33.140Z", - "EndTime": "2026-07-10T11:20:41.660Z", - "CreationTime": "2026-07-10T11:20:33.140Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/78f94b6048.json deleted file mode 100644 index f71a080fc8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/README.md deleted file mode 100644 index 3626260c3f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Classic Invoke Workflow File — Runtime Argument Mismatch (Stale Key) - -Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. - -## What this scenario exercises - -A job faults at an `Invoke Workflow File` with `System.ArgumentException` — "the following keys from the -input dictionary do not map to arguments and must be removed: in_Amount". The agent must recognize -**argument-name drift**: the parent passes `in_Amount`, but the invoked `ValidatePayment.xaml` renamed -that input to `in_GrossAmount`. The invoke fails at argument binding, before the child body runs. Fix: -Import Arguments and remap to `in_GrossAmount`. - -Distinct from the design-time `classic-invoke-arg-required` scenario (a Studio validation error about an -*unmapped required* argument): here the parent maps a *stale key that no longer exists* on the child, -and the failure is a **runtime** `ArgumentException`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` invoking `ValidatePayment.xaml` with `in_Amount`; the child declares `in_GrossAmount` | -| `data/m/r/*.json` | faulted-job fixtures (`jobs get`/`logs`/`history`/`traces`, `folders`/`jobs list`) carrying the argument-binding `ArgumentException` | -| `data/m/r/manifest.json` | `docsai ask` passthrough + job/folder rules + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the stale-argument-key mismatch and the fix (Import Arguments, remap to - `in_GrossAmount`). - -Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` (runtime -"argument name/type/direction mismatch" branch). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/e2a5346a88.json deleted file mode 100644 index 911c6f484e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-16T14:41:08.900Z", - "Message": "Invoke ValidatePayment: The values provided for the root activity's arguments did not satisfy the root activity's requirements: The following keys from the input dictionary do not map to arguments and must be removed: in_Amount. Please note that argument names are case sensitive.\r\nParameter name: rootArgumentValues\r\nin Main.xaml\r\n at InvokeWorkflowFile \"Invoke ValidatePayment\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentException\r\n at System.Activities.WorkflowApplication.EnsureInitialized()" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-16T14:41:08.400Z", - "Message": "PaymentValidation execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/manifest.json deleted file mode 100644 index 1aac3ff25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs traces", - "file": "3672254a58.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..6cd1f0ad36 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4e9d2a71-6c05-4f38-9b12-8a3f0e7c1d59\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PaymentValidation\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-16T14:41:08.220Z\",\r\n \"EndTime\": \"2026-07-16T14:41:09.010Z\",\r\n \"CreationTime\": \"2026-07-16T14:41:08.220Z\"\r\n },\r\n {\r\n \"Key\": \"9b0e4c22-1a76-4d38-8f05-2c7a1e9b6d43\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"PaymentValidation\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-10T11:20:33.140Z\",\r\n \"EndTime\": \"2026-07-10T11:20:41.660Z\",\r\n \"CreationTime\": \"2026-07-10T11:20:33.140Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-16T14:41:08.900Z\",\r\n \"Message\": \"Invoke ValidatePayment: The values provided for the root activity's arguments did not satisfy the root activity's requirements: The following keys from the input dictionary do not map to arguments and must be removed: in_Amount. Please note that argument names are case sensitive.\\r\\nParameter name: rootArgumentValues\\r\\nin Main.xaml\\r\\n at InvokeWorkflowFile \\\"Invoke ValidatePayment\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentException\\r\\n at System.Activities.WorkflowApplication.EnsureInitialized()\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-16T14:41:08.400Z\",\r\n \"Message\": \"PaymentValidation execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-16T14:41:09.010Z\",\r\n \"JobId\": 66123980\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-16T14:41:08.470Z\",\r\n \"JobId\": 66123980\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4e9d2a71-6c05-4f38-9b12-8a3f0e7c1d59\",\r\n \"StartTime\": \"2026-07-16T14:41:08.220Z\",\r\n \"EndTime\": \"2026-07-16T14:41:09.010Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"Info\": \"Invoke ValidatePayment: The values provided for the root activity's arguments did not satisfy the root activity's requirements: The following keys from the input dictionary do not map to arguments and must be removed: in_Amount. Please note that argument names are case sensitive.\\r\\nParameter name: rootArgumentValues\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeWorkflowFile \\\"Invoke ValidatePayment\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: The values provided for the root activity's arguments did not satisfy the root activity's requirements: The following keys from the input dictionary do not map to arguments and must be removed: in_Amount. Please note that argument names are case sensitive.\\r\\nParameter name: rootArgumentValues\\r\\n at System.Activities.Hosting.WorkflowInstance.RegisterExtensionManager(WorkflowInstanceExtensionManager extensionManager)\\r\\n at System.Activities.WorkflowApplication.EnsureInitialized()\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.d__.MoveNext()\",\r\n \"CreationTime\": \"2026-07-16T14:41:08.220Z\",\r\n \"ReleaseName\": \"PaymentValidation\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-16T14:41:09.010Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"4e9d2a716c054f389b128a3f0e7c1d59\",\r\n \"Id\": 66123980\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/task.yaml index 544cc404a0..27e158d79c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-arg-mismatch/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/README.md new file mode 100644 index 0000000000..ec66e7b546 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/README.md @@ -0,0 +1,28 @@ +# Classic Invoke Workflow File — Cache Mechanism Error / Code 7 (Design-Time) + +Design-time (Studio build) troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. + +## What this scenario exercises + +Studio reports `Invoked workflows are missing` + `Cache Mechanism Error (Error code: 7)` at build/debug +time, even though the invoked child `.xaml` files are present in the project. The agent must recognize +this as **project cache corruption** (not a real missing dependency) and prescribe clearing the cache. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` invoking two children, both `ValidateOrder.xaml` and `PostOrder.xaml` present, plus a stale `obj/` build artifact | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified cache corruption and the fix (close Studio, delete `.local`/`bin`/`obj`, reopen). + +Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-file-design-time-errors.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/m/r/README.md deleted file mode 100644 index 0de69cceeb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Classic Invoke Workflow File — Cache Mechanism Error / Code 7 (Design-Time) - -Design-time (Studio build) troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. - -## What this scenario exercises - -Studio reports `Invoked workflows are missing` + `Cache Mechanism Error (Error code: 7)` at build/debug -time, even though the invoked child `.xaml` files are present in the project. The agent must recognize -this as **project cache corruption** (not a real missing dependency) and prescribe clearing the cache. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` invoking two children, both `ValidateOrder.xaml` and `PostOrder.xaml` present, plus a stale `obj/` build artifact | -| `data/m/r/manifest.json` | `docsai ask` passthrough + permissive empty `unmocked_default` — no job/trace exists for a build-time error | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified cache corruption and the fix (close Studio, delete `.local`/`bin`/`obj`, reopen). - -Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-file-design-time-errors.md`. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/m/r/manifest.json deleted file mode 100644 index 8d4a4a10b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/m/r/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. Only docsai is proxied to the real CLI; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/uip-fixture.json new file mode 100644 index 0000000000..bcb5501faa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/data/uip-fixture.json @@ -0,0 +1,8 @@ +{ + "version": 1, + "responses": [], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/task.yaml index accd0d2c4d..bced371f40 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-cache-error-7/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/README.md new file mode 100644 index 0000000000..a79de292b5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/README.md @@ -0,0 +1,32 @@ +# Classic Invoke Workflow File — Child Workflow Throws (Runtime Propagation) + +Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. + +## What this scenario exercises + +A job faults with `System.NullReferenceException` at an `Invoke Workflow File`. The real fault is +**inside the invoked child** (`ProcessPayments.xaml`): a `List` variable `lineItems` is used +(`.Sum()`) while null. The agent must trace **one hop into the child** and attribute the root cause +there — the invoke is only the propagation point — rather than blaming the Invoke Workflow File itself. +This tests the skill's causal-precedence / one-hop-upstream rule. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` invoking `ProcessPayments.xaml`; the child dereferences an uninitialized `lineItems` list | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent attributed the fault to the child workflow's uninitialized-list NRE (not the invoke) and fixed + the child. + +Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` (runtime +"child workflow threw" branch). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/244ab94cfe.json deleted file mode 100644 index 141f823416..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-16T09:14:23.640Z", - "JobId": 66120441 - }, - { - "State": "Running", - "Timestamp": "2026-07-16T09:14:22.360Z", - "JobId": 66120441 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/3672254a58.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/3672254a58.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/3672254a58.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/464db476bb.json deleted file mode 100644 index a5139edf06..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/464db476bb.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7a3f1e08-2b9c-4d51-8e6a-1f0c4b2d9e33", - "StartTime": "2026-07-16T09:14:22.101Z", - "EndTime": "2026-07-16T09:14:23.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "Info": "Object reference not set to an instance of an object.\r\n\r\nin ProcessPayments.xaml\r\n at Assign \"Sum line items\"\r\n at Sequence \"ProcessPayments\"\r\n at InvokeWorkflowFile \"Invoke ProcessPayments\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at lambda_method(Closure , ActivityContext )\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContextUntyped(ActivityInstance parentInstance, ActivityWithResult expressionActivity)\r\n at UiPath.Core.Activities.InvokeWorkflowFile.<>c__DisplayClass.b__0()\r\n at UiPath.Core.Activities.InvokeWorkflowFile.EndExecute(NativeActivityContext context, IAsyncResult result)", - "CreationTime": "2026-07-16T09:14:22.101Z", - "ReleaseName": "PaymentRun", - "Type": "Attended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "b7c93a10-2d54-4e88-9a16-3f0e7c2b5d41", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-16T09:14:23.640Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7a3f1e082b9c4d518e6a1f0c4b2d9e33", - "Id": 66120441 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/6059da6ae1.json deleted file mode 100644 index 08517a2881..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7a3f1e08-2b9c-4d51-8e6a-1f0c4b2d9e33", - "State": "Faulted", - "ProcessName": "PaymentRun", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-16T09:14:22.101Z", - "EndTime": "2026-07-16T09:14:23.640Z", - "CreationTime": "2026-07-16T09:14:22.101Z" - }, - { - "Key": "1d5b0c39-8e42-4a77-b013-6c9f2a1e7d80", - "State": "Successful", - "ProcessName": "PaymentRun", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-15T16:02:11.500Z", - "EndTime": "2026-07-15T16:02:19.220Z", - "CreationTime": "2026-07-15T16:02:11.500Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/78f94b6048.json deleted file mode 100644 index 4296ab05ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "5f2a9c81-6b03-4d27-8e14-9a0f3c7b2d56", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "b7c93a10-2d54-4e88-9a16-3f0e7c2b5d41", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/README.md deleted file mode 100644 index 0894277cb0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Classic Invoke Workflow File — Child Workflow Throws (Runtime Propagation) - -Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. - -## What this scenario exercises - -A job faults with `System.NullReferenceException` at an `Invoke Workflow File`. The real fault is -**inside the invoked child** (`ProcessPayments.xaml`): a `List` variable `lineItems` is used -(`.Sum()`) while null. The agent must trace **one hop into the child** and attribute the root cause -there — the invoke is only the propagation point — rather than blaming the Invoke Workflow File itself. -This tests the skill's causal-precedence / one-hop-upstream rule. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` invoking `ProcessPayments.xaml`; the child dereferences an uninitialized `lineItems` list | -| `data/m/r/*.json` | faulted-job fixtures (`jobs get`/`logs`/`history`/`traces`, `folders`/`jobs list`) whose stack localizes the fault inside the child, below the InvokeWorkflowFile frame | -| `data/m/r/manifest.json` | `docsai ask` passthrough + job/folder rules + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent attributed the fault to the child workflow's uninitialized-list NRE (not the invoke) and fixed - the child. - -Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` (runtime -"child workflow threw" branch). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/e2a5346a88.json deleted file mode 100644 index be45240d9b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-16T09:14:23.510Z", - "Message": "Invoke ProcessPayments: Object reference not set to an instance of an object.\r\nin ProcessPayments.xaml\r\n at Assign \"Sum line items\"\r\n at Sequence \"ProcessPayments\"\r\n at InvokeWorkflowFile \"Invoke ProcessPayments\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.NullReferenceException\r\n at lambda_method(Closure , ActivityContext )\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-16T09:14:22.300Z", - "Message": "PaymentRun execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/manifest.json deleted file mode 100644 index 1aac3ff25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs traces", - "file": "3672254a58.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/uip-fixture.json new file mode 100644 index 0000000000..87bd52c017 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"5f2a9c81-6b03-4d27-8e14-9a0f3c7b2d56\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b7c93a10-2d54-4e88-9a16-3f0e7c2b5d41\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7a3f1e08-2b9c-4d51-8e6a-1f0c4b2d9e33\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PaymentRun\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-16T09:14:22.101Z\",\r\n \"EndTime\": \"2026-07-16T09:14:23.640Z\",\r\n \"CreationTime\": \"2026-07-16T09:14:22.101Z\"\r\n },\r\n {\r\n \"Key\": \"1d5b0c39-8e42-4a77-b013-6c9f2a1e7d80\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"PaymentRun\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-15T16:02:11.500Z\",\r\n \"EndTime\": \"2026-07-15T16:02:19.220Z\",\r\n \"CreationTime\": \"2026-07-15T16:02:11.500Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-16T09:14:23.510Z\",\r\n \"Message\": \"Invoke ProcessPayments: Object reference not set to an instance of an object.\\r\\nin ProcessPayments.xaml\\r\\n at Assign \\\"Sum line items\\\"\\r\\n at Sequence \\\"ProcessPayments\\\"\\r\\n at InvokeWorkflowFile \\\"Invoke ProcessPayments\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at lambda_method(Closure , ActivityContext )\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-16T09:14:22.300Z\",\r\n \"Message\": \"PaymentRun execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-16T09:14:23.640Z\",\r\n \"JobId\": 66120441\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-16T09:14:22.360Z\",\r\n \"JobId\": 66120441\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7a3f1e08-2b9c-4d51-8e6a-1f0c4b2d9e33\",\r\n \"StartTime\": \"2026-07-16T09:14:22.101Z\",\r\n \"EndTime\": \"2026-07-16T09:14:23.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"Info\": \"Object reference not set to an instance of an object.\\r\\n\\r\\nin ProcessPayments.xaml\\r\\n at Assign \\\"Sum line items\\\"\\r\\n at Sequence \\\"ProcessPayments\\\"\\r\\n at InvokeWorkflowFile \\\"Invoke ProcessPayments\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at lambda_method(Closure , ActivityContext )\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContextUntyped(ActivityInstance parentInstance, ActivityWithResult expressionActivity)\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.<>c__DisplayClass.b__0()\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.EndExecute(NativeActivityContext context, IAsyncResult result)\",\r\n \"CreationTime\": \"2026-07-16T09:14:22.101Z\",\r\n \"ReleaseName\": \"PaymentRun\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"b7c93a10-2d54-4e88-9a16-3f0e7c2b5d41\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-16T09:14:23.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7a3f1e082b9c4d518e6a1f0c4b2d9e33\",\r\n \"Id\": 66120441\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/task.yaml index ef7b575191..0abf29ca95 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-child-throws/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/README.md new file mode 100644 index 0000000000..4f4bc2767f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/README.md @@ -0,0 +1,34 @@ +# Classic Invoke Workflow File — Invoked Workflow Excluded from Package (Runtime FileNotFound) + +Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. + +## What this scenario exercises + +The process runs in Studio but a robot job faults with `System.IO.FileNotFoundException` at the Invoke +Workflow File. The invoked `Helpers\CalculateTax.xaml` is present in the project source but listed in +`project.json` `designOptions.processOptions.ignoredFiles`, so it is excluded from the published +package and missing on the robot. The agent must find the `ignoredFiles` exclusion and fix it. + +Distinct from the design-time `classic-invoke-file-not-packed` scenario (invoked `.xaml` **outside** the +project root): here the file is **inside** the project but **explicitly excluded** from packing, and the +failure surfaces only at **run time** on the robot. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` invokes `Helpers\CalculateTax.xaml` (present); `project.json` lists that file in `ignoredFiles` | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the `ignoredFiles` package exclusion and the fix (remove it, re-publish). + +Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` +(workflow-file-missing / not-packed branch). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/244ab94cfe.json deleted file mode 100644 index 09e2557499..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-16T13:22:06.220Z", - "JobId": 66122640 - }, - { - "State": "Running", - "Timestamp": "2026-07-16T13:22:05.520Z", - "JobId": 66122640 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/3672254a58.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/3672254a58.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/3672254a58.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/464db476bb.json deleted file mode 100644 index 0ecf1d57be..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/464db476bb.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3c8f1a52-7b09-4d64-9e28-1a5c0f3e7b26", - "StartTime": "2026-07-16T13:22:05.410Z", - "EndTime": "2026-07-16T13:22:06.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Invoke CalculateTax: Could not find file 'C:\\Users\\robot\\.nuget\\packages\\taxprocessing\\1.0.22\\lib\\net6.0\\Helpers\\CalculateTax.xaml'.\r\n\r\nin Main.xaml\r\n at InvokeWorkflowFile \"Invoke CalculateTax\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\nSystem.IO.FileNotFoundException: Could not find file 'Helpers\\CalculateTax.xaml'.\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.FileStream..ctor(String path, FileMode mode)\r\n at UiPath.Core.Activities.InvokeWorkflowFile.LoadWorkflow(String fileName)\r\n at UiPath.Core.Activities.InvokeWorkflowFile.d__.MoveNext()", - "CreationTime": "2026-07-16T13:22:05.410Z", - "ReleaseName": "TaxProcessing", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "d4e08b31-5f29-4c48-a163-8a1d2c9f7e60", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-16T13:22:06.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3c8f1a527b094d649e281a5c0f3e7b26", - "Id": 66122640 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/6059da6ae1.json deleted file mode 100644 index 16ecb4861f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3c8f1a52-7b09-4d64-9e28-1a5c0f3e7b26", - "State": "Faulted", - "ProcessName": "TaxProcessing", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-16T13:22:05.410Z", - "EndTime": "2026-07-16T13:22:06.220Z", - "CreationTime": "2026-07-16T13:22:05.410Z" - }, - { - "Key": "5a2c0e93-8f14-4b67-9d02-3e7a1c5f9b48", - "State": "Successful", - "ProcessName": "TaxProcessing", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DEV-STUDIO-01", - "StartTime": "2026-07-14T08:11:02.330Z", - "EndTime": "2026-07-14T08:11:07.880Z", - "CreationTime": "2026-07-14T08:11:02.330Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/78f94b6048.json deleted file mode 100644 index f8b83d846a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "9d2e6b04-8c15-4f37-a049-1b7f3c0e8d59", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "d4e08b31-5f29-4c48-a163-8a1d2c9f7e60", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/README.md deleted file mode 100644 index 1c293b2923..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# Classic Invoke Workflow File — Invoked Workflow Excluded from Package (Runtime FileNotFound) - -Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. - -## What this scenario exercises - -The process runs in Studio but a robot job faults with `System.IO.FileNotFoundException` at the Invoke -Workflow File. The invoked `Helpers\CalculateTax.xaml` is present in the project source but listed in -`project.json` `designOptions.processOptions.ignoredFiles`, so it is excluded from the published -package and missing on the robot. The agent must find the `ignoredFiles` exclusion and fix it. - -Distinct from the design-time `classic-invoke-file-not-packed` scenario (invoked `.xaml` **outside** the -project root): here the file is **inside** the project but **explicitly excluded** from packing, and the -failure surfaces only at **run time** on the robot. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` invokes `Helpers\CalculateTax.xaml` (present); `project.json` lists that file in `ignoredFiles` | -| `data/m/r/*.json` | faulted-job fixtures (`jobs get`/`logs`/`history`/`traces`, `folders`/`jobs list`) carrying the runtime `FileNotFoundException` at `InvokeWorkflowFile.LoadWorkflow` | -| `data/m/r/manifest.json` | `docsai ask` passthrough + job/folder rules + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the `ignoredFiles` package exclusion and the fix (remove it, re-publish). - -Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` -(workflow-file-missing / not-packed branch). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/e2a5346a88.json deleted file mode 100644 index 48184f931b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-16T13:22:06.100Z", - "Message": "Invoke CalculateTax: Could not find file 'Helpers\\CalculateTax.xaml'.\r\nin Main.xaml\r\n at InvokeWorkflowFile \"Invoke CalculateTax\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IO.FileNotFoundException\r\n at UiPath.Core.Activities.InvokeWorkflowFile.LoadWorkflow(String fileName)" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-16T13:22:05.600Z", - "Message": "TaxProcessing execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/manifest.json deleted file mode 100644 index 1aac3ff25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs traces", - "file": "3672254a58.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/uip-fixture.json new file mode 100644 index 0000000000..ca78ad3549 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d2e6b04-8c15-4f37-a049-1b7f3c0e8d59\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"d4e08b31-5f29-4c48-a163-8a1d2c9f7e60\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3c8f1a52-7b09-4d64-9e28-1a5c0f3e7b26\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TaxProcessing\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-16T13:22:05.410Z\",\r\n \"EndTime\": \"2026-07-16T13:22:06.220Z\",\r\n \"CreationTime\": \"2026-07-16T13:22:05.410Z\"\r\n },\r\n {\r\n \"Key\": \"5a2c0e93-8f14-4b67-9d02-3e7a1c5f9b48\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"TaxProcessing\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DEV-STUDIO-01\",\r\n \"StartTime\": \"2026-07-14T08:11:02.330Z\",\r\n \"EndTime\": \"2026-07-14T08:11:07.880Z\",\r\n \"CreationTime\": \"2026-07-14T08:11:02.330Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-16T13:22:06.100Z\",\r\n \"Message\": \"Invoke CalculateTax: Could not find file 'Helpers\\\\CalculateTax.xaml'.\\r\\nin Main.xaml\\r\\n at InvokeWorkflowFile \\\"Invoke CalculateTax\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.LoadWorkflow(String fileName)\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-16T13:22:05.600Z\",\r\n \"Message\": \"TaxProcessing execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-16T13:22:06.220Z\",\r\n \"JobId\": 66122640\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-16T13:22:05.520Z\",\r\n \"JobId\": 66122640\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3c8f1a52-7b09-4d64-9e28-1a5c0f3e7b26\",\r\n \"StartTime\": \"2026-07-16T13:22:05.410Z\",\r\n \"EndTime\": \"2026-07-16T13:22:06.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Invoke CalculateTax: Could not find file 'C:\\\\Users\\\\robot\\\\.nuget\\\\packages\\\\taxprocessing\\\\1.0.22\\\\lib\\\\net6.0\\\\Helpers\\\\CalculateTax.xaml'.\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeWorkflowFile \\\"Invoke CalculateTax\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Could not find file 'Helpers\\\\CalculateTax.xaml'.\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.FileStream..ctor(String path, FileMode mode)\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.LoadWorkflow(String fileName)\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.d__.MoveNext()\",\r\n \"CreationTime\": \"2026-07-16T13:22:05.410Z\",\r\n \"ReleaseName\": \"TaxProcessing\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"d4e08b31-5f29-4c48-a163-8a1d2c9f7e60\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-16T13:22:06.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3c8f1a527b094d649e281a5c0f3e7b26\",\r\n \"Id\": 66122640\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/task.yaml index 9d2fdc7815..ba653220eb 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-ignored-file/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/README.md new file mode 100644 index 0000000000..76df0c6d7a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/README.md @@ -0,0 +1,32 @@ +# Classic Invoke Workflow File — Persistence Not Supported (Runtime) + +Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. + +## What this scenario exercises + +A robot job faults with `System.NotSupportedException` — the invoked workflow requires persistence but +the runtime does not support it. `SubmitForApproval.xaml` runs a `Run Job` in **Suspend** mode +(persistence required), while `project.json` has `runtimeOptions.supportsPersistence: false`. The agent +must connect the suspending activity to the missing persistence support and fix it (enable +`supportsPersistence`, or switch `Run Job` out of Suspend mode). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` invokes `SubmitForApproval.xaml`, which runs `RunJob ExecutionMode="Suspend"`; `project.json` has `supportsPersistence: false` | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the persistence-not-supported cause and the fix (`supportsPersistence: true`, or + drop Suspend mode). + +Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` +(persistence branch). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/244ab94cfe.json deleted file mode 100644 index 1129200a0d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-16T15:18:34.560Z", - "JobId": 66124315 - }, - { - "State": "Running", - "Timestamp": "2026-07-16T15:18:33.810Z", - "JobId": 66124315 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/3672254a58.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/3672254a58.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/3672254a58.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/464db476bb.json deleted file mode 100644 index b4983f0391..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/464db476bb.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8d2c5f91-4a06-4e73-9b18-3c7f0a2e6d45", - "StartTime": "2026-07-16T15:18:33.720Z", - "EndTime": "2026-07-16T15:18:34.560Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Run approval routing job: Persistence is not supported in the current runtime. The workflow attempted to suspend (Run Job in Suspend mode), which requires persistence, but the process is not configured for persistence.\r\n\r\nin SubmitForApproval.xaml\r\n at RunJob \"Run approval routing job\"\r\n at Sequence \"SubmitForApproval\"\r\n at InvokeWorkflowFile \"Invoke SubmitForApproval\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\nSystem.NotSupportedException: This workflow instance requires persistence, which is not supported in the current runtime.\r\n at System.Activities.WorkflowApplication.EnsurePersistenceProvider()\r\n at UiPath.Core.Activities.RunJob.d__.MoveNext()", - "CreationTime": "2026-07-16T15:18:33.720Z", - "ReleaseName": "OrderApproval", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "e5f19c42-6a38-4b57-a274-9b2d3c0f8e71", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-16T15:18:34.560Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8d2c5f914a064e739b183c7f0a2e6d45", - "Id": 66124315 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/6059da6ae1.json deleted file mode 100644 index a89247de25..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "8d2c5f91-4a06-4e73-9b18-3c7f0a2e6d45", - "State": "Faulted", - "ProcessName": "OrderApproval", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-16T15:18:33.720Z", - "EndTime": "2026-07-16T15:18:34.560Z", - "CreationTime": "2026-07-16T15:18:33.720Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/78f94b6048.json deleted file mode 100644 index f2e77083a8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "a1e07c53-9d28-4f69-b031-2a6f5c0e8d47", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "e5f19c42-6a38-4b57-a274-9b2d3c0f8e71", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/README.md deleted file mode 100644 index 425bf45cd5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Classic Invoke Workflow File — Persistence Not Supported (Runtime) - -Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. - -## What this scenario exercises - -A robot job faults with `System.NotSupportedException` — the invoked workflow requires persistence but -the runtime does not support it. `SubmitForApproval.xaml` runs a `Run Job` in **Suspend** mode -(persistence required), while `project.json` has `runtimeOptions.supportsPersistence: false`. The agent -must connect the suspending activity to the missing persistence support and fix it (enable -`supportsPersistence`, or switch `Run Job` out of Suspend mode). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` invokes `SubmitForApproval.xaml`, which runs `RunJob ExecutionMode="Suspend"`; `project.json` has `supportsPersistence: false` | -| `data/m/r/*.json` | faulted-job fixtures (`jobs get`/`logs`/`history`/`traces`, `folders`/`jobs list`) carrying the persistence `NotSupportedException` at the child's `RunJob` | -| `data/m/r/manifest.json` | `docsai ask` passthrough + job/folder rules + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the persistence-not-supported cause and the fix (`supportsPersistence: true`, or - drop Suspend mode). - -Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` -(persistence branch). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/e2a5346a88.json deleted file mode 100644 index a9f8ba4b88..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-16T15:18:34.440Z", - "Message": "Run approval routing job: Persistence is not supported in the current runtime. The workflow attempted to suspend (Run Job in Suspend mode), which requires persistence, but the process is not configured for persistence.\r\nin SubmitForApproval.xaml\r\n at RunJob \"Run approval routing job\"\r\n at Sequence \"SubmitForApproval\"\r\n at InvokeWorkflowFile \"Invoke SubmitForApproval\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.NotSupportedException\r\n at UiPath.Core.Activities.RunJob.d__.MoveNext()" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-16T15:18:33.900Z", - "Message": "OrderApproval execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/manifest.json deleted file mode 100644 index 1aac3ff25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs traces", - "file": "3672254a58.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/uip-fixture.json new file mode 100644 index 0000000000..65518a2f1c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1e07c53-9d28-4f69-b031-2a6f5c0e8d47\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e5f19c42-6a38-4b57-a274-9b2d3c0f8e71\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8d2c5f91-4a06-4e73-9b18-3c7f0a2e6d45\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderApproval\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-16T15:18:33.720Z\",\r\n \"EndTime\": \"2026-07-16T15:18:34.560Z\",\r\n \"CreationTime\": \"2026-07-16T15:18:33.720Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-16T15:18:34.440Z\",\r\n \"Message\": \"Run approval routing job: Persistence is not supported in the current runtime. The workflow attempted to suspend (Run Job in Suspend mode), which requires persistence, but the process is not configured for persistence.\\r\\nin SubmitForApproval.xaml\\r\\n at RunJob \\\"Run approval routing job\\\"\\r\\n at Sequence \\\"SubmitForApproval\\\"\\r\\n at InvokeWorkflowFile \\\"Invoke SubmitForApproval\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NotSupportedException\\r\\n at UiPath.Core.Activities.RunJob.d__.MoveNext()\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-16T15:18:33.900Z\",\r\n \"Message\": \"OrderApproval execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-16T15:18:34.560Z\",\r\n \"JobId\": 66124315\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-16T15:18:33.810Z\",\r\n \"JobId\": 66124315\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8d2c5f91-4a06-4e73-9b18-3c7f0a2e6d45\",\r\n \"StartTime\": \"2026-07-16T15:18:33.720Z\",\r\n \"EndTime\": \"2026-07-16T15:18:34.560Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Run approval routing job: Persistence is not supported in the current runtime. The workflow attempted to suspend (Run Job in Suspend mode), which requires persistence, but the process is not configured for persistence.\\r\\n\\r\\nin SubmitForApproval.xaml\\r\\n at RunJob \\\"Run approval routing job\\\"\\r\\n at Sequence \\\"SubmitForApproval\\\"\\r\\n at InvokeWorkflowFile \\\"Invoke SubmitForApproval\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nSystem.NotSupportedException: This workflow instance requires persistence, which is not supported in the current runtime.\\r\\n at System.Activities.WorkflowApplication.EnsurePersistenceProvider()\\r\\n at UiPath.Core.Activities.RunJob.d__.MoveNext()\",\r\n \"CreationTime\": \"2026-07-16T15:18:33.720Z\",\r\n \"ReleaseName\": \"OrderApproval\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"e5f19c42-6a38-4b57-a274-9b2d3c0f8e71\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-16T15:18:34.560Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8d2c5f914a064e739b183c7f0a2e6d45\",\r\n \"Id\": 66124315\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/task.yaml index 11d5af3d94..16126a1da7 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-persistence/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/README.md new file mode 100644 index 0000000000..2626b4798c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/README.md @@ -0,0 +1,31 @@ +# Classic Invoke Workflow File — Invalid Isolated / Target-Session Combination (Runtime) + +Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. + +## What this scenario exercises + +A job faults with `System.Activities.InvalidWorkflowException` at the Invoke Workflow File's +`CacheMetadata`. The invoke sets `TargetSession="PictureInPicture"` (non-`Current`) while `UnSafe` +(Isolated) is `False` — a combination the runtime rejects (a non-`Current` session requires isolated +execution). The agent must identify the invalid session/isolation combo and fix it (`UnSafe=True`, or +revert `TargetSession` to `Current`). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` invokes `GenerateReport.xaml` with `UnSafe="False"` + `TargetSession="PictureInPicture"` | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the invalid `TargetSession`/`UnSafe` combination and the fix. + +Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` +(isolated/elevated/target-session branch). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/244ab94cfe.json deleted file mode 100644 index d7bcf8552b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-16T11:03:48.190Z", - "JobId": 66121775 - }, - { - "State": "Running", - "Timestamp": "2026-07-16T11:03:47.700Z", - "JobId": 66121775 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/3672254a58.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/3672254a58.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/3672254a58.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/464db476bb.json deleted file mode 100644 index 295893ebd2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/464db476bb.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "6b1e4d70-3a92-4c58-8f01-2d7c9a3e5b14", - "StartTime": "2026-07-16T11:03:47.610Z", - "EndTime": "2026-07-16T11:03:48.190Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "Info": "Invoke GenerateReport: Validation Error: When TargetSession is set to a value other than Current, the invoked workflow must run isolated. TargetSession 'PictureInPicture' requires UnSafe (Isolated) = True.\r\n\r\nin Main.xaml\r\n at InvokeWorkflowFile \"Invoke GenerateReport\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\nSystem.Activities.InvalidWorkflowException: Validation errors encountered while processing the workflow tree.\r\n at System.Activities.Validation.ActivityValidationServices.ThrowIfViolationsExist(IList`1 validationErrors)\r\n at UiPath.Core.Activities.InvokeWorkflowFile.CacheMetadata(NativeActivityMetadata metadata)\r\n at UiPath.Core.Activities.InvokeWorkflowFile.d__.MoveNext()", - "CreationTime": "2026-07-16T11:03:47.610Z", - "ReleaseName": "ReportDispatch", - "Type": "Attended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "c3d97a20-4e18-4b37-9a52-7f0c1b8e6d49", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-16T11:03:48.190Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "6b1e4d703a924c588f012d7c9a3e5b14", - "Id": 66121775 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/6059da6ae1.json deleted file mode 100644 index 15b8cb495e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "6b1e4d70-3a92-4c58-8f01-2d7c9a3e5b14", - "State": "Faulted", - "ProcessName": "ReportDispatch", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-16T11:03:47.610Z", - "EndTime": "2026-07-16T11:03:48.190Z", - "CreationTime": "2026-07-16T11:03:47.610Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/78f94b6048.json deleted file mode 100644 index 8f47477ac3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "8c1f5a03-7d24-4e69-9b31-0a6f2c9e4d78", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "c3d97a20-4e18-4b37-9a52-7f0c1b8e6d49", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/README.md deleted file mode 100644 index 5fea35756f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/README.md +++ /dev/null @@ -1,30 +0,0 @@ -# Classic Invoke Workflow File — Invalid Isolated / Target-Session Combination (Runtime) - -Runtime troubleshooting scenario for `UiPath.Core.Activities.InvokeWorkflowFile`. - -## What this scenario exercises - -A job faults with `System.Activities.InvalidWorkflowException` at the Invoke Workflow File's -`CacheMetadata`. The invoke sets `TargetSession="PictureInPicture"` (non-`Current`) while `UnSafe` -(Isolated) is `False` — a combination the runtime rejects (a non-`Current` session requires isolated -execution). The agent must identify the invalid session/isolation combo and fix it (`UnSafe=True`, or -revert `TargetSession` to `Current`). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` invokes `GenerateReport.xaml` with `UnSafe="False"` + `TargetSession="PictureInPicture"` | -| `data/m/r/*.json` | faulted-job fixtures (`jobs get`/`logs`/`history`/`traces`, `folders`/`jobs list`) carrying the validation error at `InvokeWorkflowFile.CacheMetadata` | -| `data/m/r/manifest.json` | `docsai ask` passthrough + job/folder rules + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the invalid `TargetSession`/`UnSafe` combination and the fix. - -Playbook: `references/activity-packages/classic-activities/playbooks/invoke-workflow-failed.md` -(isolated/elevated/target-session branch). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/e2a5346a88.json deleted file mode 100644 index 5978d4bd37..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-16T11:03:48.080Z", - "Message": "Invoke GenerateReport: Validation Error: When TargetSession is set to a value other than Current, the invoked workflow must run isolated. TargetSession 'PictureInPicture' requires UnSafe (Isolated) = True.\r\nin Main.xaml\r\n at InvokeWorkflowFile \"Invoke GenerateReport\"\r\n at Sequence \"Main\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Activities.InvalidWorkflowException\r\n at UiPath.Core.Activities.InvokeWorkflowFile.CacheMetadata(NativeActivityMetadata metadata)" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-16T11:03:47.760Z", - "Message": "ReportDispatch execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/manifest.json deleted file mode 100644 index 1aac3ff25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs traces", - "file": "3672254a58.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/uip-fixture.json new file mode 100644 index 0000000000..72230a4db5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8c1f5a03-7d24-4e69-9b31-0a6f2c9e4d78\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c3d97a20-4e18-4b37-9a52-7f0c1b8e6d49\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6b1e4d70-3a92-4c58-8f01-2d7c9a3e5b14\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDispatch\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-16T11:03:47.610Z\",\r\n \"EndTime\": \"2026-07-16T11:03:48.190Z\",\r\n \"CreationTime\": \"2026-07-16T11:03:47.610Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-16T11:03:48.080Z\",\r\n \"Message\": \"Invoke GenerateReport: Validation Error: When TargetSession is set to a value other than Current, the invoked workflow must run isolated. TargetSession 'PictureInPicture' requires UnSafe (Isolated) = True.\\r\\nin Main.xaml\\r\\n at InvokeWorkflowFile \\\"Invoke GenerateReport\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Activities.InvalidWorkflowException\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.CacheMetadata(NativeActivityMetadata metadata)\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-16T11:03:47.760Z\",\r\n \"Message\": \"ReportDispatch execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-16T11:03:48.190Z\",\r\n \"JobId\": 66121775\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-16T11:03:47.700Z\",\r\n \"JobId\": 66121775\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"6b1e4d70-3a92-4c58-8f01-2d7c9a3e5b14\",\r\n \"StartTime\": \"2026-07-16T11:03:47.610Z\",\r\n \"EndTime\": \"2026-07-16T11:03:48.190Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"Info\": \"Invoke GenerateReport: Validation Error: When TargetSession is set to a value other than Current, the invoked workflow must run isolated. TargetSession 'PictureInPicture' requires UnSafe (Isolated) = True.\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeWorkflowFile \\\"Invoke GenerateReport\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nSystem.Activities.InvalidWorkflowException: Validation errors encountered while processing the workflow tree.\\r\\n at System.Activities.Validation.ActivityValidationServices.ThrowIfViolationsExist(IList`1 validationErrors)\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.CacheMetadata(NativeActivityMetadata metadata)\\r\\n at UiPath.Core.Activities.InvokeWorkflowFile.d__.MoveNext()\",\r\n \"CreationTime\": \"2026-07-16T11:03:47.610Z\",\r\n \"ReleaseName\": \"ReportDispatch\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"c3d97a20-4e18-4b37-9a52-7f0c1b8e6d49\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-16T11:03:48.190Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"6b1e4d703a924c588f012d7c9a3e5b14\",\r\n \"Id\": 66121775\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/task.yaml index 1bb7a86925..e624e99031 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-invoke-session-isolated/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/README.md new file mode 100644 index 0000000000..4b0c683ea4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/README.md @@ -0,0 +1,19 @@ +# Classic MoveFile Source Not Found — de-DE localized canary + +Locale-robustness canary for the grep-based playbook routing design: identical scenario to +`classic-movefile-source-not-found`, but every agent-visible .NET framework message in the +fixtures is localized to German (de-DE robot host), as real non-English Windows hosts emit. +Exception class names (`System.IO.FileNotFoundException`) and stack frames stay invariant, +matching real .NET behavior. + +What it validates: routing must succeed via language-invariant signals (exception class) and/or +translate-before-grep of the localized message — per SKILL.md's localized-error-text rule. +A failure here means the skill's diagnosis depends on English message text. + +CLI-only variant: the project source snapshot is deliberately omitted so the ONLY evidence path runs through the localized runtime fixtures — a source snapshot lets agents shortcut to design-time evidence and never touch the German text (observed in the first validation run). + +Grading: identical contract to the English variant (`RESOLUTION.md` — same root cause, same fix). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/0ac831386e.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/0ac831386e.json deleted file mode 100644 index b8fa4af470..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/0ac831386e.json +++ /dev/null @@ -1,390 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T12:57:51.543Z", - "EndTime": "2026-06-04T12:57:52.790Z", - "CreationTime": "2026-06-04T12:57:51.543Z" - }, - { - "Key": "35c669cb-7f46-4282-b90d-ca7abd52c59f", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:50:06.750Z", - "EndTime": "2026-06-04T10:50:12.563Z", - "CreationTime": "2026-06-04T10:50:06.750Z" - }, - { - "Key": "b1cab28d-5ab7-42f1-98f3-1920b637ea06", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:42:10.817Z", - "EndTime": "2026-06-04T10:42:43.897Z", - "CreationTime": "2026-06-04T10:42:10.817Z" - }, - { - "Key": "8e2c5979-e05f-4e03-b6d1-cda02aa8450c", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T15:10:44.610Z", - "EndTime": "2026-06-03T15:11:17.967Z", - "CreationTime": "2026-06-03T15:10:44.610Z" - }, - { - "Key": "cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T11:56:44.780Z", - "EndTime": "2026-06-03T11:56:52.850Z", - "CreationTime": "2026-06-03T11:56:44.780Z" - }, - { - "Key": "025d0847-ce15-494f-91d6-04be2a19be6d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:28:08.133Z", - "EndTime": "2026-06-03T09:29:10.690Z", - "CreationTime": "2026-06-03T09:28:08.133Z" - }, - { - "Key": "90058dec-f287-4b4d-8272-1b7cf72e0c0d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:17:53.650Z", - "EndTime": "2026-06-03T09:18:57.633Z", - "CreationTime": "2026-06-03T09:17:53.650Z" - }, - { - "Key": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T07:30:38.093Z", - "EndTime": "2026-06-02T07:31:37.367Z", - "CreationTime": "2026-06-02T07:30:38.093Z" - }, - { - "Key": "68d80da7-719f-44cd-b7d6-94be679f55d4", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:52:06.580Z", - "EndTime": "2026-06-02T06:52:40.293Z", - "CreationTime": "2026-06-02T06:52:06.580Z" - }, - { - "Key": "3975787c-87d9-4927-bc7c-a19165e06035", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:42:21.737Z", - "EndTime": "2026-06-02T06:42:56.063Z", - "CreationTime": "2026-06-02T06:42:21.737Z" - }, - { - "Key": "ccc3e841-bdcf-4501-9d30-f112f098f630", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T13:09:18.353Z", - "EndTime": "2026-05-28T13:10:10.310Z", - "CreationTime": "2026-05-28T12:48:07.547Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 27, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/1afc4ab735.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/1afc4ab735.json deleted file mode 100644 index 3f725b1394..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/1afc4ab735.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "StartTime": "2026-06-04T12:57:51.543Z", - "EndTime": "2026-06-04T12:57:52.790Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "Info": "Die Datei \"C:\\UiPathDemo\\DoesNotExist\\source.txt\" konnte nicht gefunden werden.\r\n\r\nin move.xaml\r\n at MoveFile \"Move File\"\r\n at Sequence \"move\"\r\n at DynamicActivity \"move\"\r\n\r\nSystem.IO.FileNotFoundException: Die Datei \"C:\\UiPathDemo\\DoesNotExist\\source.txt\" konnte nicht gefunden werden.\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\r\n at System.IO.File.Move(String sourceFileName, String destFileName)\r\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-04T12:57:51.543Z", - "ReleaseName": "Legacy", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20819368, - "EntryPointPath": "move.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T12:57:52.790Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f32f9829446549a28e3e0cc182a020c4", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 66080537 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/README.md deleted file mode 100644 index 0b07649826..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# Classic MoveFile Source Not Found — de-DE localized canary - -Locale-robustness canary for the grep-based playbook routing design: identical scenario to -`classic-movefile-source-not-found`, but every agent-visible .NET framework message in the -fixtures is localized to German (de-DE robot host), as real non-English Windows hosts emit. -Exception class names (`System.IO.FileNotFoundException`) and stack frames stay invariant, -matching real .NET behavior. - -What it validates: routing must succeed via language-invariant signals (exception class) and/or -translate-before-grep of the localized message — per SKILL.md's localized-error-text rule. -A failure here means the skill's diagnosis depends on English message text. - -CLI-only variant: the project source snapshot is deliberately omitted so the ONLY evidence path runs through the localized runtime fixtures — a source snapshot lets agents shortcut to design-time evidence and never touch the German text (observed in the first validation run). - -Grading: identical contract to the English variant (`RESOLUTION.md` — same root cause, same fix). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/c9afc88bba.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/c9afc88bba.json deleted file mode 100644 index 4aff279c30..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/c9afc88bba.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T12:57:52.579Z", - "Message": "Move File: Die Datei \"C:\\UiPathDemo\\DoesNotExist\\source.txt\" konnte nicht gefunden werden.\r\nin move.xaml\r\n at MoveFile \"Move File\"\r\n at Sequence \"move\"\r\n at DynamicActivity \"move\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IO.FileNotFoundException\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\r\n at System.IO.File.Move(String sourceFileName, String destFileName)\r\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/caaaec7c69.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/caaaec7c69.json deleted file mode 100644 index d66b160f18..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/caaaec7c69.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-04T12:57:53.120Z", - "JobId": 66080537 - }, - { - "State": "Running", - "Timestamp": "2026-06-04T12:57:51.810Z", - "JobId": 66080537 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/manifest.json deleted file mode 100644 index 51ca183acc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Fixtures rebuilt verbatim from the real investigation raw/ captures (job f32f9829, process Legacy, folder Shared). Scrubbed: real email->original_email@test.com, host->MOCK-HOST. C:\\UiPathDemo paths preserved (root-cause evidence). Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language queries (2 distinct); proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "525de3e550.json" - }, - { - "match": "or jobs list", - "file": "0ac831386e.json" - }, - { - "match": "or jobs logs", - "file": "c9afc88bba.json" - }, - { - "match": "or jobs history", - "file": "caaaec7c69.json" - }, - { - "match": "or jobs traces", - "file": "2d930e2440.json" - }, - { - "match": "or jobs get", - "file": "1afc4ab735.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/uip-fixture.json new file mode 100644 index 0000000000..d4cbd7a516 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"EndTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"CreationTime\": \"2026-06-04T12:57:51.543Z\"\r\n },\r\n {\r\n \"Key\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"EndTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"CreationTime\": \"2026-06-04T10:50:06.750Z\"\r\n },\r\n {\r\n \"Key\": \"b1cab28d-5ab7-42f1-98f3-1920b637ea06\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:42:10.817Z\",\r\n \"EndTime\": \"2026-06-04T10:42:43.897Z\",\r\n \"CreationTime\": \"2026-06-04T10:42:10.817Z\"\r\n },\r\n {\r\n \"Key\": \"8e2c5979-e05f-4e03-b6d1-cda02aa8450c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T15:10:44.610Z\",\r\n \"EndTime\": \"2026-06-03T15:11:17.967Z\",\r\n \"CreationTime\": \"2026-06-03T15:10:44.610Z\"\r\n },\r\n {\r\n \"Key\": \"cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T11:56:44.780Z\",\r\n \"EndTime\": \"2026-06-03T11:56:52.850Z\",\r\n \"CreationTime\": \"2026-06-03T11:56:44.780Z\"\r\n },\r\n {\r\n \"Key\": \"025d0847-ce15-494f-91d6-04be2a19be6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:28:08.133Z\",\r\n \"EndTime\": \"2026-06-03T09:29:10.690Z\",\r\n \"CreationTime\": \"2026-06-03T09:28:08.133Z\"\r\n },\r\n {\r\n \"Key\": \"90058dec-f287-4b4d-8272-1b7cf72e0c0d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:17:53.650Z\",\r\n \"EndTime\": \"2026-06-03T09:18:57.633Z\",\r\n \"CreationTime\": \"2026-06-03T09:17:53.650Z\"\r\n },\r\n {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\"\r\n },\r\n {\r\n \"Key\": \"68d80da7-719f-44cd-b7d6-94be679f55d4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:52:06.580Z\",\r\n \"EndTime\": \"2026-06-02T06:52:40.293Z\",\r\n \"CreationTime\": \"2026-06-02T06:52:06.580Z\"\r\n },\r\n {\r\n \"Key\": \"3975787c-87d9-4927-bc7c-a19165e06035\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:42:21.737Z\",\r\n \"EndTime\": \"2026-06-02T06:42:56.063Z\",\r\n \"CreationTime\": \"2026-06-02T06:42:21.737Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 27,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T12:57:52.579Z\",\r\n \"Message\": \"Move File: Die Datei \\\"C:\\\\UiPathDemo\\\\DoesNotExist\\\\source.txt\\\" konnte nicht gefunden werden.\\r\\nin move.xaml\\r\\n at MoveFile \\\"Move File\\\"\\r\\n at Sequence \\\"move\\\"\\r\\n at DynamicActivity \\\"move\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\\r\\n at System.IO.File.Move(String sourceFileName, String destFileName)\\r\\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-04T12:57:53.120Z\",\r\n \"JobId\": 66080537\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-04T12:57:51.810Z\",\r\n \"JobId\": 66080537\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"StartTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"EndTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"Info\": \"Die Datei \\\"C:\\\\UiPathDemo\\\\DoesNotExist\\\\source.txt\\\" konnte nicht gefunden werden.\\r\\n\\r\\nin move.xaml\\r\\n at MoveFile \\\"Move File\\\"\\r\\n at Sequence \\\"move\\\"\\r\\n at DynamicActivity \\\"move\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Die Datei \\\"C:\\\\UiPathDemo\\\\DoesNotExist\\\\source.txt\\\" konnte nicht gefunden werden.\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\\r\\n at System.IO.File.Move(String sourceFileName, String destFileName)\\r\\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"ReleaseName\": \"Legacy\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20819368,\r\n \"EntryPointPath\": \"move.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f32f9829446549a28e3e0cc182a020c4\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 66080537\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/task.yaml index a057c24f63..312c3e5547 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found-de/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/README.md new file mode 100644 index 0000000000..0e8b1c58c8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/README.md @@ -0,0 +1,36 @@ +# Classic Movefile Source Not Found — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The classic **Move File** activity in the **Legacy** process points at a hardcoded source path that does not exist on the robot machine, so the job faulted the moment that activity ran. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name classic-movefile-source-not-found --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/0ac831386e.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/0ac831386e.json deleted file mode 100644 index b8fa4af470..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/0ac831386e.json +++ /dev/null @@ -1,390 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T12:57:51.543Z", - "EndTime": "2026-06-04T12:57:52.790Z", - "CreationTime": "2026-06-04T12:57:51.543Z" - }, - { - "Key": "35c669cb-7f46-4282-b90d-ca7abd52c59f", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:50:06.750Z", - "EndTime": "2026-06-04T10:50:12.563Z", - "CreationTime": "2026-06-04T10:50:06.750Z" - }, - { - "Key": "b1cab28d-5ab7-42f1-98f3-1920b637ea06", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:42:10.817Z", - "EndTime": "2026-06-04T10:42:43.897Z", - "CreationTime": "2026-06-04T10:42:10.817Z" - }, - { - "Key": "8e2c5979-e05f-4e03-b6d1-cda02aa8450c", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T15:10:44.610Z", - "EndTime": "2026-06-03T15:11:17.967Z", - "CreationTime": "2026-06-03T15:10:44.610Z" - }, - { - "Key": "cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T11:56:44.780Z", - "EndTime": "2026-06-03T11:56:52.850Z", - "CreationTime": "2026-06-03T11:56:44.780Z" - }, - { - "Key": "025d0847-ce15-494f-91d6-04be2a19be6d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:28:08.133Z", - "EndTime": "2026-06-03T09:29:10.690Z", - "CreationTime": "2026-06-03T09:28:08.133Z" - }, - { - "Key": "90058dec-f287-4b4d-8272-1b7cf72e0c0d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:17:53.650Z", - "EndTime": "2026-06-03T09:18:57.633Z", - "CreationTime": "2026-06-03T09:17:53.650Z" - }, - { - "Key": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T07:30:38.093Z", - "EndTime": "2026-06-02T07:31:37.367Z", - "CreationTime": "2026-06-02T07:30:38.093Z" - }, - { - "Key": "68d80da7-719f-44cd-b7d6-94be679f55d4", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:52:06.580Z", - "EndTime": "2026-06-02T06:52:40.293Z", - "CreationTime": "2026-06-02T06:52:06.580Z" - }, - { - "Key": "3975787c-87d9-4927-bc7c-a19165e06035", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:42:21.737Z", - "EndTime": "2026-06-02T06:42:56.063Z", - "CreationTime": "2026-06-02T06:42:21.737Z" - }, - { - "Key": "ccc3e841-bdcf-4501-9d30-f112f098f630", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T13:09:18.353Z", - "EndTime": "2026-05-28T13:10:10.310Z", - "CreationTime": "2026-05-28T12:48:07.547Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 27, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/1afc4ab735.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/1afc4ab735.json deleted file mode 100644 index 423b70c056..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/1afc4ab735.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "StartTime": "2026-06-04T12:57:51.543Z", - "EndTime": "2026-06-04T12:57:52.790Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "Info": "Could not find file 'C:\\UiPathDemo\\DoesNotExist\\source.txt'.\r\n\r\nin move.xaml\r\n at MoveFile \"Move File\"\r\n at Sequence \"move\"\r\n at DynamicActivity \"move\"\r\n\r\nSystem.IO.FileNotFoundException: Could not find file 'C:\\UiPathDemo\\DoesNotExist\\source.txt'.\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\r\n at System.IO.File.Move(String sourceFileName, String destFileName)\r\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-04T12:57:51.543Z", - "ReleaseName": "Legacy", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20819368, - "EntryPointPath": "move.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T12:57:52.790Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f32f9829446549a28e3e0cc182a020c4", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 66080537 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/README.md deleted file mode 100644 index ecce4a6984..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Classic Movefile Source Not Found — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The classic **Move File** activity in the **Legacy** process points at a hardcoded source path that does not exist on the robot machine, so the job faulted the moment that activity ran. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name classic-movefile-source-not-found --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/c9afc88bba.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/c9afc88bba.json deleted file mode 100644 index b6f22e968f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/c9afc88bba.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T12:57:52.579Z", - "Message": "Move File: Could not find file 'C:\\UiPathDemo\\DoesNotExist\\source.txt'.\r\nin move.xaml\r\n at MoveFile \"Move File\"\r\n at Sequence \"move\"\r\n at DynamicActivity \"move\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IO.FileNotFoundException\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\r\n at System.IO.File.Move(String sourceFileName, String destFileName)\r\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/caaaec7c69.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/caaaec7c69.json deleted file mode 100644 index d66b160f18..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/caaaec7c69.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-04T12:57:53.120Z", - "JobId": 66080537 - }, - { - "State": "Running", - "Timestamp": "2026-06-04T12:57:51.810Z", - "JobId": 66080537 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/manifest.json deleted file mode 100644 index 51ca183acc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Fixtures rebuilt verbatim from the real investigation raw/ captures (job f32f9829, process Legacy, folder Shared). Scrubbed: real email->original_email@test.com, host->MOCK-HOST. C:\\UiPathDemo paths preserved (root-cause evidence). Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language queries (2 distinct); proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "525de3e550.json" - }, - { - "match": "or jobs list", - "file": "0ac831386e.json" - }, - { - "match": "or jobs logs", - "file": "c9afc88bba.json" - }, - { - "match": "or jobs history", - "file": "caaaec7c69.json" - }, - { - "match": "or jobs traces", - "file": "2d930e2440.json" - }, - { - "match": "or jobs get", - "file": "1afc4ab735.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..913f841a6a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"EndTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"CreationTime\": \"2026-06-04T12:57:51.543Z\"\r\n },\r\n {\r\n \"Key\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"EndTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"CreationTime\": \"2026-06-04T10:50:06.750Z\"\r\n },\r\n {\r\n \"Key\": \"b1cab28d-5ab7-42f1-98f3-1920b637ea06\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:42:10.817Z\",\r\n \"EndTime\": \"2026-06-04T10:42:43.897Z\",\r\n \"CreationTime\": \"2026-06-04T10:42:10.817Z\"\r\n },\r\n {\r\n \"Key\": \"8e2c5979-e05f-4e03-b6d1-cda02aa8450c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T15:10:44.610Z\",\r\n \"EndTime\": \"2026-06-03T15:11:17.967Z\",\r\n \"CreationTime\": \"2026-06-03T15:10:44.610Z\"\r\n },\r\n {\r\n \"Key\": \"cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T11:56:44.780Z\",\r\n \"EndTime\": \"2026-06-03T11:56:52.850Z\",\r\n \"CreationTime\": \"2026-06-03T11:56:44.780Z\"\r\n },\r\n {\r\n \"Key\": \"025d0847-ce15-494f-91d6-04be2a19be6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:28:08.133Z\",\r\n \"EndTime\": \"2026-06-03T09:29:10.690Z\",\r\n \"CreationTime\": \"2026-06-03T09:28:08.133Z\"\r\n },\r\n {\r\n \"Key\": \"90058dec-f287-4b4d-8272-1b7cf72e0c0d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:17:53.650Z\",\r\n \"EndTime\": \"2026-06-03T09:18:57.633Z\",\r\n \"CreationTime\": \"2026-06-03T09:17:53.650Z\"\r\n },\r\n {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\"\r\n },\r\n {\r\n \"Key\": \"68d80da7-719f-44cd-b7d6-94be679f55d4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:52:06.580Z\",\r\n \"EndTime\": \"2026-06-02T06:52:40.293Z\",\r\n \"CreationTime\": \"2026-06-02T06:52:06.580Z\"\r\n },\r\n {\r\n \"Key\": \"3975787c-87d9-4927-bc7c-a19165e06035\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:42:21.737Z\",\r\n \"EndTime\": \"2026-06-02T06:42:56.063Z\",\r\n \"CreationTime\": \"2026-06-02T06:42:21.737Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 27,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T12:57:52.579Z\",\r\n \"Message\": \"Move File: Could not find file 'C:\\\\UiPathDemo\\\\DoesNotExist\\\\source.txt'.\\r\\nin move.xaml\\r\\n at MoveFile \\\"Move File\\\"\\r\\n at Sequence \\\"move\\\"\\r\\n at DynamicActivity \\\"move\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\\r\\n at System.IO.File.Move(String sourceFileName, String destFileName)\\r\\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-04T12:57:53.120Z\",\r\n \"JobId\": 66080537\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-04T12:57:51.810Z\",\r\n \"JobId\": 66080537\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"StartTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"EndTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"Info\": \"Could not find file 'C:\\\\UiPathDemo\\\\DoesNotExist\\\\source.txt'.\\r\\n\\r\\nin move.xaml\\r\\n at MoveFile \\\"Move File\\\"\\r\\n at Sequence \\\"move\\\"\\r\\n at DynamicActivity \\\"move\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Could not find file 'C:\\\\UiPathDemo\\\\DoesNotExist\\\\source.txt'.\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)\\r\\n at System.IO.File.Move(String sourceFileName, String destFileName)\\r\\n at UiPath.Core.Activities.MoveFile.d__25.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"ReleaseName\": \"Legacy\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20819368,\r\n \"EntryPointPath\": \"move.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f32f9829446549a28e3e0cc182a020c4\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 66080537\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/task.yaml index 9bf415a443..fe4b254b17 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-movefile-source-not-found/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/README.md new file mode 100644 index 0000000000..8c3e95c7b2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/README.md @@ -0,0 +1,36 @@ +# Classic Openapp Wrong Scope Type — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The classic UI Automation scope `Open Application 'msedge.exe Google'` in `Workflow.xaml` is the wrong scope type for its target. It is a **desktop** Open Application (launching `chrome.exe`) but was given a **browser-style** html selector with no application window, so it never produces a usable context window. Its child `Click 'push button'` faults the instant the scope hands it an uninitialized window — before the Click's own (valid) selector is ever evaluated. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name classic-openapp-wrong-scope-type --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/75efc2982c.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/75efc2982c.json deleted file mode 100644 index 390a22e56f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/75efc2982c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-04T10:50:12.593Z", - "JobId": 66068165 - }, - { - "State": "Running", - "Timestamp": "2026-06-04T10:50:07.033Z", - "JobId": 66068165 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/README.md deleted file mode 100644 index 73d92548ed..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Classic Openapp Wrong Scope Type — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The classic UI Automation scope `Open Application 'msedge.exe Google'` in `Workflow.xaml` is the wrong scope type for its target. It is a **desktop** Open Application (launching `chrome.exe`) but was given a **browser-style** html selector with no application window, so it never produces a usable context window. Its child `Click 'push button'` faults the instant the scope hands it an uninitialized window — before the Click's own (valid) selector is ever evaluated. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name classic-openapp-wrong-scope-type --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/b9e9a4f3d1.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/b9e9a4f3d1.json deleted file mode 100644 index 2a4d394715..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/b9e9a4f3d1.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "35c669cb-7f46-4282-b90d-ca7abd52c59f", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:50:06.750Z", - "EndTime": "2026-06-04T10:50:12.563Z", - "CreationTime": "2026-06-04T10:50:06.750Z" - }, - { - "Key": "b1cab28d-5ab7-42f1-98f3-1920b637ea06", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:42:10.817Z", - "EndTime": "2026-06-04T10:42:43.897Z", - "CreationTime": "2026-06-04T10:42:10.817Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/dc417814f6.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/dc417814f6.json deleted file mode 100644 index 9fde5d93d1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/dc417814f6.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "35c669cb-7f46-4282-b90d-ca7abd52c59f", - "StartTime": "2026-06-04T10:50:06.750Z", - "EndTime": "2026-06-04T10:50:12.563Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "35c669cb-7f46-4282-b90d-ca7abd52c59f", - "Info": "The target Element was not specified for this activity. You should set its Target property or use this activity inside of a scope activity (Attach Browser, Open Browser, Open Application, Attach Window, Get Active Window, Element Scope activities).\r\n\r\nin Workflow.xaml\r\n at Click \"Click 'push button'\"\r\n at Sequence \"Do\"\r\n at OpenApplication \"Open Application 'msedge.exe Google'\"\r\n at Sequence \"Workflow\"\r\n at DynamicActivity \"Workflow\"\r\n\r\nUiPath.Core.Activities.ElementNotSetException: The target Element was not specified for this activity. You should set its Target property or use this activity inside of a scope activity (Attach Browser, Open Browser, Open Application, Attach Window, Get Active Window, Element Scope activities).\r\n ---> UiPath.Core.UninitializedNodeException: The UiElement is not initialized\r\n at UiPath.Core.UiElement.ValidateSearchArgs(FindScope scope, Selector selector)\r\n at UiPath.Core.UiElement.FindFirst(FindScope scope, Selector selector, IPlaceholderResolver placeholderResolver)\r\n at UiPath.Core.Activities.Target.FindByElement(UiElement targetElement, String fromSelector, IPlaceholderResolver placeholderResolver)\r\n at UiPath.Core.Activities.Target.<>c__DisplayClass58_0.b__2()\r\n at UiPath.Core.Activities.Target.<>c__DisplayClass58_0.b__4()\r\n at UiPath.Core.Activities.Target.GetElement(Func`1 findMethod, Nullable`1 delayBefore, Region clippingRegion, Boolean highlight, Boolean isClippingRegionNormalized)\r\n at UiPath.Core.Activities.Target.<>c__DisplayClass56_0.b__0()\r\n at System.Threading.Tasks.Task`1.InnerInvoke()\r\n at System.Threading.Tasks.Task.Execute()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at UiPath.Core.Activities.Target.d__56.MoveNext()\r\n --- End of inner exception stack trace ---\r\n at UiPath.Core.Activities.ScopeActivity.OnFaulted(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.Invoke(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-04T10:50:06.750Z", - "ReleaseName": "Legacy", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20816350, - "EntryPointPath": "Workflow.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T10:50:12.563Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "35c669cb7f464282b90dca7abd52c59f", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 66068165 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/f620e553ca.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/f620e553ca.json deleted file mode 100644 index 1fec2c02f1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/f620e553ca.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T10:50:12.351Z", - "Message": "Click 'push button': The target Element was not specified for this activity. You should set its Target property or use this activity inside of a scope activity (Attach Browser, Open Browser, Open Application, Attach Window, Get Active Window, Element Scope activities).\r\nin Workflow.xaml\r\n at Click \"Click 'push button'\"\r\n at Sequence \"Do\"\r\n at OpenApplication \"Open Application 'msedge.exe Google'\"\r\n at Sequence \"Workflow\"\r\n at DynamicActivity \"Workflow\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.Core.Activities.ElementNotSetException\r\n at UiPath.Core.Activities.ScopeActivity.OnFaulted(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.Invoke(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/manifest.json deleted file mode 100644 index 9c25f9c5b2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Fixtures rebuilt verbatim from the real investigation raw/ captures (job 35c669cb, process Legacy, folder Shared). Scrubbed: real email->original_email@test.com, host->MOCK-HOST. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "525de3e550.json" - }, - { - "match": "or jobs list", - "file": "b9e9a4f3d1.json" - }, - { - "match": "or jobs logs", - "file": "f620e553ca.json" - }, - { - "match": "or jobs history", - "file": "75efc2982c.json" - }, - { - "match": "or jobs get", - "file": "dc417814f6.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/uip-fixture.json new file mode 100644 index 0000000000..89ddd48ee0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"EndTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"CreationTime\": \"2026-06-04T10:50:06.750Z\"\r\n },\r\n {\r\n \"Key\": \"b1cab28d-5ab7-42f1-98f3-1920b637ea06\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:42:10.817Z\",\r\n \"EndTime\": \"2026-06-04T10:42:43.897Z\",\r\n \"CreationTime\": \"2026-06-04T10:42:10.817Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T10:50:12.351Z\",\r\n \"Message\": \"Click 'push button': The target Element was not specified for this activity. You should set its Target property or use this activity inside of a scope activity (Attach Browser, Open Browser, Open Application, Attach Window, Get Active Window, Element Scope activities).\\r\\nin Workflow.xaml\\r\\n at Click \\\"Click 'push button'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at OpenApplication \\\"Open Application 'msedge.exe Google'\\\"\\r\\n at Sequence \\\"Workflow\\\"\\r\\n at DynamicActivity \\\"Workflow\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.Core.Activities.ElementNotSetException\\r\\n at UiPath.Core.Activities.ScopeActivity.OnFaulted(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.Invoke(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-04T10:50:12.593Z\",\r\n \"JobId\": 66068165\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-04T10:50:07.033Z\",\r\n \"JobId\": 66068165\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"StartTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"EndTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"Info\": \"The target Element was not specified for this activity. You should set its Target property or use this activity inside of a scope activity (Attach Browser, Open Browser, Open Application, Attach Window, Get Active Window, Element Scope activities).\\r\\n\\r\\nin Workflow.xaml\\r\\n at Click \\\"Click 'push button'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at OpenApplication \\\"Open Application 'msedge.exe Google'\\\"\\r\\n at Sequence \\\"Workflow\\\"\\r\\n at DynamicActivity \\\"Workflow\\\"\\r\\n\\r\\nUiPath.Core.Activities.ElementNotSetException: The target Element was not specified for this activity. You should set its Target property or use this activity inside of a scope activity (Attach Browser, Open Browser, Open Application, Attach Window, Get Active Window, Element Scope activities).\\r\\n ---> UiPath.Core.UninitializedNodeException: The UiElement is not initialized\\r\\n at UiPath.Core.UiElement.ValidateSearchArgs(FindScope scope, Selector selector)\\r\\n at UiPath.Core.UiElement.FindFirst(FindScope scope, Selector selector, IPlaceholderResolver placeholderResolver)\\r\\n at UiPath.Core.Activities.Target.FindByElement(UiElement targetElement, String fromSelector, IPlaceholderResolver placeholderResolver)\\r\\n at UiPath.Core.Activities.Target.<>c__DisplayClass58_0.b__2()\\r\\n at UiPath.Core.Activities.Target.<>c__DisplayClass58_0.b__4()\\r\\n at UiPath.Core.Activities.Target.GetElement(Func`1 findMethod, Nullable`1 delayBefore, Region clippingRegion, Boolean highlight, Boolean isClippingRegionNormalized)\\r\\n at UiPath.Core.Activities.Target.<>c__DisplayClass56_0.b__0()\\r\\n at System.Threading.Tasks.Task`1.InnerInvoke()\\r\\n at System.Threading.Tasks.Task.Execute()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at UiPath.Core.Activities.Target.d__56.MoveNext()\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Core.Activities.ScopeActivity.OnFaulted(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.Invoke(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"ReleaseName\": \"Legacy\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20816350,\r\n \"EntryPointPath\": \"Workflow.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"35c669cb7f464282b90dca7abd52c59f\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 66068165\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/task.yaml index 3b18356e12..623246d023 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-openapp-wrong-scope-type/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/README.md new file mode 100644 index 0000000000..fab5b9b994 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/README.md @@ -0,0 +1,42 @@ +# classic-runlocaltriggers-generated-missing + +Faithful-replay scenario for the `uipath-troubleshoot` skill. + +## What this scenario covers + +An attended `OrderEntryAssistant` automation (folder `Operations`) faults ~2 seconds after start. The +project uses **Run Local Triggers**, whose read-only trigger workflow Studio auto-generates in the +hidden design-time cache at `.local\generated\Triggers.Generated.xaml`. That generated file was **not +present** when the package the robot ran was built — the `.local` cache is not normally committed, so a +cloned/shared project (or a stale cache / a Studio↔Assistant version mismatch after an upgrade) lacks +it. At run time Run Local Triggers cannot load the generated workflow and the job faults. + +The Error log is **deliberately generic** — `System.Exception: The invoked workflow could not be loaded. +A referenced workflow file was not found in the running package.` at `RunLocalTriggers.Execute` → +`GeneratedWorkflowLoader.Load` — it does **not** name the file. So the diagnosis cannot be read straight +off the log: the agent has to combine the faulting activity (`Run Local Triggers`), the project layout +(no `.local` generated workflow), and the knowledge that Run Local Triggers loads an auto-generated +trigger workflow. This mirrors the `classic-openapp` design (misleading surface, cause established by +investigation) so the scenario reliably exercises the skill rather than being one-shot from the error. + +This is a **signature** case (Faulted job with an Error log). The agent routes via +`classic-activities/summary.md` to +`classic-activities/playbooks/trigger-scope-and-local-triggers-failed.md`. + +The correct diagnosis names the **missing auto-generated Run Local Triggers workflow** +(`.local\generated\Triggers.Generated.xaml`), and the fix **regenerate it by opening/rebuilding the +project in Studio and republishing** (delete the stale `.local` cache first if corrupt; align +Studio/Assistant versions if the fault followed an upgrade) — never hand-create the file. Wrong turns to +avoid: misreading the generic "invoked workflow could not be loaded" as an `Invoke Workflow File` +path/dependency bug (there is no Invoke Workflow File in the source), a workflow-authoring bug in +`Main.xaml`, or fixing it by writing the file by hand. + +## Evidence layout + +- `process/` — the failing project source: `Main.xaml` has a `ui:RunLocalTriggers` activity, `project.json` depends on `UiPath.Form.Activities`, and there is **no `.local` folder** (the missing artifact is the cause) + +Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/0061763428.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/0061763428.json deleted file mode 100644 index 46d4d1159c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/0061763428.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63", - "StartTime": "2026-07-21T14:02:11.310Z", - "EndTime": "2026-07-21T14:02:13.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63", - "Info": "Job faulted at startup while starting local triggers.", - "CreationTime": "2026-07-21T14:02:10.980Z", - "ReleaseName": "OrderEntryAssistant", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Attended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 61044188, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 771204, - "FolderKey": "d8f1a3c5-2b64-4e90-a1f5-7c9e0d3b6a42", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "CONTOSO\\aturner", - "OrchestratorUserIdentity": "aturner@contoso.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-21T14:02:13.640Z", - "Id": 91044501 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/4d0407f78c.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/4d0407f78c.json deleted file mode 100644 index 39523aa628..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/4d0407f78c.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "Message": "OrderEntryAssistant execution error: System.Exception: The invoked workflow could not be loaded. A referenced workflow file was not found in the running package.\n at UiPath.Studio.LocalTriggers.GeneratedWorkflowLoader.Load(String path)\n at UiPath.Core.Activities.RunLocalTriggers.Execute(NativeActivityContext context)\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivity(...)", - "TimeStamp": "2026-07-21T14:02:13.520Z", - "RobotName": "aturner", - "HostMachineName": "MOCK-HOST", - "ProcessName": "OrderEntryAssistant" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/6c3e3e03c3.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/6c3e3e03c3.json deleted file mode 100644 index 6c48740a81..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/6c3e3e03c3.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "Message": "OrderEntryAssistant execution started", - "TimeStamp": "2026-07-21T14:02:11.520Z", - "RobotName": "aturner", - "HostMachineName": "MOCK-HOST", - "ProcessName": "OrderEntryAssistant" - }, - { - "Level": "Trace", - "Message": "Log assistant ready execution started", - "TimeStamp": "2026-07-21T14:02:12.040Z", - "RobotName": "aturner", - "ProcessName": "OrderEntryAssistant" - }, - { - "Level": "Information", - "Message": "Order Entry Assistant ready", - "TimeStamp": "2026-07-21T14:02:12.120Z", - "RobotName": "aturner", - "HostMachineName": "MOCK-HOST", - "ProcessName": "OrderEntryAssistant" - }, - { - "Level": "Trace", - "Message": "Run Local Triggers execution started", - "TimeStamp": "2026-07-21T14:02:12.900Z", - "RobotName": "aturner", - "ProcessName": "OrderEntryAssistant" - }, - { - "Level": "Trace", - "Message": "Run Local Triggers execution ended (faulted)", - "TimeStamp": "2026-07-21T14:02:13.560Z", - "RobotName": "aturner", - "ProcessName": "OrderEntryAssistant" - } - ], - "Pagination": { - "Returned": 5, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/75949d659e.json deleted file mode 100644 index 8812477c5a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/75949d659e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63", - "State": "Faulted", - "ProcessName": "OrderEntryAssistant", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-21T14:02:11.310Z", - "EndTime": "2026-07-21T14:02:13.640Z", - "CreationTime": "2026-07-21T14:02:10.980Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/README.md deleted file mode 100644 index 9bef00329e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# classic-runlocaltriggers-generated-missing - -Faithful-replay scenario for the `uipath-troubleshoot` skill. - -## What this scenario covers - -An attended `OrderEntryAssistant` automation (folder `Operations`) faults ~2 seconds after start. The -project uses **Run Local Triggers**, whose read-only trigger workflow Studio auto-generates in the -hidden design-time cache at `.local\generated\Triggers.Generated.xaml`. That generated file was **not -present** when the package the robot ran was built — the `.local` cache is not normally committed, so a -cloned/shared project (or a stale cache / a Studio↔Assistant version mismatch after an upgrade) lacks -it. At run time Run Local Triggers cannot load the generated workflow and the job faults. - -The Error log is **deliberately generic** — `System.Exception: The invoked workflow could not be loaded. -A referenced workflow file was not found in the running package.` at `RunLocalTriggers.Execute` → -`GeneratedWorkflowLoader.Load` — it does **not** name the file. So the diagnosis cannot be read straight -off the log: the agent has to combine the faulting activity (`Run Local Triggers`), the project layout -(no `.local` generated workflow), and the knowledge that Run Local Triggers loads an auto-generated -trigger workflow. This mirrors the `classic-openapp` design (misleading surface, cause established by -investigation) so the scenario reliably exercises the skill rather than being one-shot from the error. - -This is a **signature** case (Faulted job with an Error log). The agent routes via -`classic-activities/summary.md` to -`classic-activities/playbooks/trigger-scope-and-local-triggers-failed.md`. - -The correct diagnosis names the **missing auto-generated Run Local Triggers workflow** -(`.local\generated\Triggers.Generated.xaml`), and the fix **regenerate it by opening/rebuilding the -project in Studio and republishing** (delete the stale `.local` cache first if corrupt; align -Studio/Assistant versions if the fault followed an upgrade) — never hand-create the file. Wrong turns to -avoid: misreading the generic "invoked workflow could not be loaded" as an `Invoke Workflow File` -path/dependency bug (there is no Invoke Workflow File in the source), a workflow-authoring bug in -`Main.xaml`, or fixing it by writing the file by hand. - -## Evidence layout - -- `data/m/r/10fold.json` — `or folders list` (Operations folder key) -- `data/m/r/20jobs.json` — `or jobs list` (the Faulted job) -- `data/m/r/30jobg.json` — `or jobs get` (State Faulted, Attended, MOCK-HOST, ~2s) -- `data/m/r/40errl.json` — `or jobs logs --level Error` (generic `invoked workflow could not be loaded` at `RunLocalTriggers.Execute` → `GeneratedWorkflowLoader.Load`; the file is **not** named) -- `data/m/r/50infl.json` — `or jobs logs` (Info/Trace: started → Run Local Triggers faulted) -- `process/` — the failing project source: `Main.xaml` has a `ui:RunLocalTriggers` activity, `project.json` depends on `UiPath.Form.Activities`, and there is **no `.local` folder** (the missing artifact is the cause) - -Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/ee2628e6b6.json deleted file mode 100644 index 87f5d4d3ca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "d8f1a3c5-2b64-4e90-a1f5-7c9e0d3b6a42", - "Name": "Operations", - "Path": "Operations", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/manifest.json deleted file mode 100644 index e9d6c867ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored fixtures for a runtime fault of a trigger-based attended automation (job 7e2a9c14, process OrderEntryAssistant, folder Operations). The job Faults ~2s after start: the project uses Run Local Triggers, whose read-only generated workflow .local\\generated\\Triggers.Generated.xaml is missing (the .local design-time cache was not present when the package was built — cloned/shared without it / stale after a Studio-Assistant version bump), so at run time Run Local Triggers cannot load the generated trigger workflow. The Error log is DELIBERATELY GENERIC ('the invoked workflow could not be loaded / a referenced workflow file was not found' at RunLocalTriggers.Execute -> GeneratedWorkflowLoader.Load) and does NOT name the file, so the diagnosis cannot be one-shot from the log: the agent must combine the faulting activity + the absent .local generated workflow + playbook knowledge (Run Local Triggers loads an auto-generated trigger workflow), and resist the Invoke-Workflow-File red herring. The process/ snapshot deliberately has NO .local folder (that missing artifact IS the cause). Matching is token-aware with plain-substring fallback (first match wins).", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs get 7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63", - "file": "0061763428.json" - }, - { - "match": "or jobs logs 7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63 --level Error", - "file": "4d0407f78c.json" - }, - { - "match": "or jobs logs 7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63", - "file": "6c3e3e03c3.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/uip-fixture.json new file mode 100644 index 0000000000..ab6aa43735 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/data/uip-fixture.json @@ -0,0 +1,64 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"Message\": \"OrderEntryAssistant execution error: System.Exception: The invoked workflow could not be loaded. A referenced workflow file was not found in the running package.\\n at UiPath.Studio.LocalTriggers.GeneratedWorkflowLoader.Load(String path)\\n at UiPath.Core.Activities.RunLocalTriggers.Execute(NativeActivityContext context)\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivity(...)\",\r\n \"TimeStamp\": \"2026-07-21T14:02:13.520Z\",\r\n \"RobotName\": \"aturner\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"OrderEntryAssistant\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63\",\r\n \"StartTime\": \"2026-07-21T14:02:11.310Z\",\r\n \"EndTime\": \"2026-07-21T14:02:13.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63\",\r\n \"Info\": \"Job faulted at startup while starting local triggers.\",\r\n \"CreationTime\": \"2026-07-21T14:02:10.980Z\",\r\n \"ReleaseName\": \"OrderEntryAssistant\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Attended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 61044188,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 771204,\r\n \"FolderKey\": \"d8f1a3c5-2b64-4e90-a1f5-7c9e0d3b6a42\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"CONTOSO\\\\aturner\",\r\n \"OrchestratorUserIdentity\": \"aturner@contoso.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-21T14:02:13.640Z\",\r\n \"Id\": 91044501\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"OrderEntryAssistant execution started\",\r\n \"TimeStamp\": \"2026-07-21T14:02:11.520Z\",\r\n \"RobotName\": \"aturner\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"OrderEntryAssistant\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Log assistant ready execution started\",\r\n \"TimeStamp\": \"2026-07-21T14:02:12.040Z\",\r\n \"RobotName\": \"aturner\",\r\n \"ProcessName\": \"OrderEntryAssistant\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"Order Entry Assistant ready\",\r\n \"TimeStamp\": \"2026-07-21T14:02:12.120Z\",\r\n \"RobotName\": \"aturner\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"OrderEntryAssistant\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Run Local Triggers execution started\",\r\n \"TimeStamp\": \"2026-07-21T14:02:12.900Z\",\r\n \"RobotName\": \"aturner\",\r\n \"ProcessName\": \"OrderEntryAssistant\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Run Local Triggers execution ended (faulted)\",\r\n \"TimeStamp\": \"2026-07-21T14:02:13.560Z\",\r\n \"RobotName\": \"aturner\",\r\n \"ProcessName\": \"OrderEntryAssistant\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 5,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"d8f1a3c5-2b64-4e90-a1f5-7c9e0d3b6a42\",\r\n \"Name\": \"Operations\",\r\n \"Path\": \"Operations\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7e2a9c14-3b58-4d6f-a091-5c8e2f1b7d63\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderEntryAssistant\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-21T14:02:11.310Z\",\r\n \"EndTime\": \"2026-07-21T14:02:13.640Z\",\r\n \"CreationTime\": \"2026-07-21T14:02:10.980Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/task.yaml index 1bbc26ddc9..e3a280b05f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-runlocaltriggers-generated-missing/task.yaml @@ -28,22 +28,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/README.md new file mode 100644 index 0000000000..d68a98c8c1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/README.md @@ -0,0 +1,32 @@ +# classic-typeinto-field-not-cleared + +Faithful-replay scenario for the `uipath-troubleshoot` skill. + +## What this scenario covers + +A scheduled unattended `VendorOnboarding` job finishes **Successful** with zero Error logs, but the +bank account number written to the Vendor Master form is wrong: `00008891245` instead of the intended +`8891245`. The classic `Type Into 'Account Number'` (`UiPath.Core.Activities.TypeInto`) has +`EmptyField = False` and no clear-field step, so the intended text was **appended** to the field's +pre-existing value (`0000`) rather than replacing it. Classic `Type Into` has no post-write +verification, so the wrong value never faulted. + +This is the Type-Into-specific **wrong/incomplete/appended text** family — distinct from a pure silent +no-op (text DID land, just concatenated) and from the throwing paths (no `SelectorNotFoundException` / +`ElementOperationException` / `ActivityTimeoutException`). It is a **no-signature** case: the agent +routes via the top-level `summary.md` no-signature table ("Successful but the output is wrong") to +`classic-activities/playbooks/type-into-wrong-or-incomplete-text.md`. + +The correct diagnosis names the append caused by `EmptyField=False` on a pre-filled field, and the fix +`EmptyField=True` (or an explicit clear) plus a read-back verification. Wrong turns to avoid: +concluding a silent no-op, an element-not-found, or a SimulateType character drop. + +## Evidence layout + +- `process/` — the failing project source (classic `Type Into` with `EmptyField="False"`) + +Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/44a081cb26.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/44a081cb26.json deleted file mode 100644 index c89b4b365d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/44a081cb26.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40", - "StartTime": "2026-07-16T23:10:02.140Z", - "EndTime": "2026-07-16T23:10:39.610Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40", - "Info": "Job completed", - "CreationTime": "2026-07-16T23:10:01.700Z", - "ReleaseName": "VendorOnboarding", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 51882043, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 771204, - "FolderKey": "c7e3a9b1-2d64-4f80-9a15-6b8c0d2e4f31", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svc_rpa", - "OrchestratorUserIdentity": "rpa-service@contoso.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-16T23:10:39.610Z", - "Id": 90773115 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/75949d659e.json deleted file mode 100644 index f7136c5c0d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/75949d659e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40", - "State": "Successful", - "ProcessName": "VendorOnboarding", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-16T23:10:02.140Z", - "EndTime": "2026-07-16T23:10:39.610Z", - "CreationTime": "2026-07-16T23:10:01.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/75f6ad6632.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/75f6ad6632.json deleted file mode 100644 index 26238d5ebb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/75f6ad6632.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/README.md deleted file mode 100644 index f36ba2231d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# classic-typeinto-field-not-cleared - -Faithful-replay scenario for the `uipath-troubleshoot` skill. - -## What this scenario covers - -A scheduled unattended `VendorOnboarding` job finishes **Successful** with zero Error logs, but the -bank account number written to the Vendor Master form is wrong: `00008891245` instead of the intended -`8891245`. The classic `Type Into 'Account Number'` (`UiPath.Core.Activities.TypeInto`) has -`EmptyField = False` and no clear-field step, so the intended text was **appended** to the field's -pre-existing value (`0000`) rather than replacing it. Classic `Type Into` has no post-write -verification, so the wrong value never faulted. - -This is the Type-Into-specific **wrong/incomplete/appended text** family — distinct from a pure silent -no-op (text DID land, just concatenated) and from the throwing paths (no `SelectorNotFoundException` / -`ElementOperationException` / `ActivityTimeoutException`). It is a **no-signature** case: the agent -routes via the top-level `summary.md` no-signature table ("Successful but the output is wrong") to -`classic-activities/playbooks/type-into-wrong-or-incomplete-text.md`. - -The correct diagnosis names the append caused by `EmptyField=False` on a pre-filled field, and the fix -`EmptyField=True` (or an explicit clear) plus a read-back verification. Wrong turns to avoid: -concluding a silent no-op, an element-not-found, or a SimulateType character drop. - -## Evidence layout - -- `data/m/r/10fold.json` — `or folders list` (Finance folder key) -- `data/m/r/20jobs.json` — `or jobs list` (the Successful job) -- `data/m/r/30jobg.json` — `or jobs get` (State Successful, Unattended, MOCK-HOST) -- `data/m/r/40errl.json` — `or jobs logs --level Error` (empty — confirms no-signature) -- `data/m/r/50infl.json` — `or jobs logs` (Info logs: `0000` before, `00008891245` after) -- `process/` — the failing project source (classic `Type Into` with `EmptyField="False"`) - -Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/e1a227ecb5.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/e1a227ecb5.json deleted file mode 100644 index b83d336912..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/e1a227ecb5.json +++ /dev/null @@ -1,72 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "Message": "VendorOnboarding execution started", - "TimeStamp": "2026-07-16T23:10:02.300Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Trace", - "Message": "Get Text 'Current Account' execution ended", - "TimeStamp": "2026-07-16T23:10:11.020Z", - "RobotName": "svc_rpa", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Information", - "Message": "Current account on file: 0000", - "TimeStamp": "2026-07-16T23:10:11.090Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Trace", - "Message": "Type Into 'Account Number' execution started", - "TimeStamp": "2026-07-16T23:10:12.400Z", - "RobotName": "svc_rpa", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Trace", - "Message": "Type Into 'Account Number' execution ended", - "TimeStamp": "2026-07-16T23:10:14.760Z", - "RobotName": "svc_rpa", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Trace", - "Message": "Get Text 'Account After Save' execution ended", - "TimeStamp": "2026-07-16T23:10:16.510Z", - "RobotName": "svc_rpa", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Information", - "Message": "Account submitted for vendor V-40817: 00008891245", - "TimeStamp": "2026-07-16T23:10:16.590Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "VendorOnboarding" - }, - { - "Level": "Information", - "Message": "VendorOnboarding execution ended", - "TimeStamp": "2026-07-16T23:10:39.500Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "VendorOnboarding" - } - ], - "Pagination": { - "Returned": 8, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/ee2628e6b6.json deleted file mode 100644 index e3cab1deea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "c7e3a9b1-2d64-4f80-9a15-6b8c0d2e4f31", - "Name": "Finance", - "Path": "Finance", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/manifest.json deleted file mode 100644 index 2f5c637fd0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored fixtures for a no-signature classic Type Into wrong-value fault (job 9d2f4a71, process VendorOnboarding, folder Finance). Job is Successful with zero Error logs; the Info logs show the field value written = existing '0000' + intended '8891245' = '00008891245' (append because EmptyField was not set). Separate `jobs logs ... --level Error` rule returns empty so the agent confirms no-signature. Matching is token-aware with plain-substring fallback (first match wins).", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs get 9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40", - "file": "44a081cb26.json" - }, - { - "match": "or jobs logs 9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40 --level Error", - "file": "75f6ad6632.json" - }, - { - "match": "or jobs logs 9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40", - "file": "e1a227ecb5.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/uip-fixture.json new file mode 100644 index 0000000000..4717ee41dd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/data/uip-fixture.json @@ -0,0 +1,64 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40\",\r\n \"StartTime\": \"2026-07-16T23:10:02.140Z\",\r\n \"EndTime\": \"2026-07-16T23:10:39.610Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-07-16T23:10:01.700Z\",\r\n \"ReleaseName\": \"VendorOnboarding\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 51882043,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 771204,\r\n \"FolderKey\": \"c7e3a9b1-2d64-4f80-9a15-6b8c0d2e4f31\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_rpa\",\r\n \"OrchestratorUserIdentity\": \"rpa-service@contoso.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-16T23:10:39.610Z\",\r\n \"Id\": 90773115\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"VendorOnboarding execution started\",\r\n \"TimeStamp\": \"2026-07-16T23:10:02.300Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Get Text 'Current Account' execution ended\",\r\n \"TimeStamp\": \"2026-07-16T23:10:11.020Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"Current account on file: 0000\",\r\n \"TimeStamp\": \"2026-07-16T23:10:11.090Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Type Into 'Account Number' execution started\",\r\n \"TimeStamp\": \"2026-07-16T23:10:12.400Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Type Into 'Account Number' execution ended\",\r\n \"TimeStamp\": \"2026-07-16T23:10:14.760Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Get Text 'Account After Save' execution ended\",\r\n \"TimeStamp\": \"2026-07-16T23:10:16.510Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"Account submitted for vendor V-40817: 00008891245\",\r\n \"TimeStamp\": \"2026-07-16T23:10:16.590Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"VendorOnboarding execution ended\",\r\n \"TimeStamp\": \"2026-07-16T23:10:39.500Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"VendorOnboarding\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 8,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c7e3a9b1-2d64-4f80-9a15-6b8c0d2e4f31\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d2f4a71-6c83-4e15-b0a7-3f8c1e5d9b40\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"VendorOnboarding\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-16T23:10:02.140Z\",\r\n \"EndTime\": \"2026-07-16T23:10:39.610Z\",\r\n \"CreationTime\": \"2026-07-16T23:10:01.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/task.yaml index 70c26f9b34..3e6a05dcca 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/classic-typeinto-field-not-cleared/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/README.md new file mode 100644 index 0000000000..d1a268e66b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/README.md @@ -0,0 +1,44 @@ +# Append To CSV Failure - DataTable Column Structure Mismatch + +This scenario reproduces an `Append To CSV` failure caused by a **DataTable +column-structure mismatch**. The workflow builds a 2-column `DataTable` +(`OrderId`, `Customer`) and appends it to `orders.csv`, which has 3 columns +(`OrderId`, `Customer`, `Total`) — so the rows cannot be aligned and the append +faults. + +## What this scenario uncovers + +**Root Cause:** The input `DataTable` (built via the two `Add Data Column` steps) +has 2 columns; the existing `orders.csv` has 3. `Append To CSV` cannot line the +rows up with the file's columns, so it faults with a column-count mismatch. + +This maps to: +`references/activity-packages/csv-activities/playbooks/csv-datatable-structure-mismatch.md` + +The discriminator vs the other CSV playbooks: this is a **data-shape** problem, +not a `CsvHelper` `Method not found` and not a file lock. The fix is a +workflow/DataTable change (align columns), so the correct agent behavior is to +recommend matching the DataTable's structure to the file — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project that initializes a `DataTable`, adds only `OrderId` + `Customer` columns, then `Append To CSV` to `data\orders.csv` (3-column file) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `csv-datatable-structure-mismatch.md` +- Agent identified the column count/header mismatch between the input DataTable + (2 columns) and the existing CSV (3 columns) and recommended aligning the + DataTable's columns to the file (add the missing column / Build Data Table with + identical headers), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/README.md deleted file mode 100644 index 446c198c4f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Append To CSV Failure - DataTable Column Structure Mismatch - -This scenario reproduces an `Append To CSV` failure caused by a **DataTable -column-structure mismatch**. The workflow builds a 2-column `DataTable` -(`OrderId`, `Customer`) and appends it to `orders.csv`, which has 3 columns -(`OrderId`, `Customer`, `Total`) — so the rows cannot be aligned and the append -faults. - -## What this scenario uncovers - -**Root Cause:** The input `DataTable` (built via the two `Add Data Column` steps) -has 2 columns; the existing `orders.csv` has 3. `Append To CSV` cannot line the -rows up with the file's columns, so it faults with a column-count mismatch. - -This maps to: -`references/activity-packages/csv-activities/playbooks/csv-datatable-structure-mismatch.md` - -The discriminator vs the other CSV playbooks: this is a **data-shape** problem, -not a `CsvHelper` `Method not found` and not a file lock. The fix is a -workflow/DataTable change (align columns), so the correct agent behavior is to -recommend matching the DataTable's structure to the file — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project that initializes a `DataTable`, adds only `OrderId` + `Customer` columns, then `Append To CSV` to `data\orders.csv` (3-column file) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `csv-datatable-structure-mismatch.md` -- Agent identified the column count/header mismatch between the input DataTable - (2 columns) and the existing CSV (3 columns) and recommended aligning the - DataTable's columns to the file (add the missing column / Build Data Table with - identical headers), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/manifest.json deleted file mode 100644 index edf3932c89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ca030003-d4e5-4f60-8a03-000000000003\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ca030003-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ca030003-d4e5-4f60-8a03-000000000003 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ca030003-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"cb030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-get-cb030003-output-json.json" - }, - { - "match": "or jobs get cb030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-get-cb030003-output-json.json" - }, - { - "match": "or jobs logs \"cb030003-e5f6-4071-9b03-000000000003\" --level Error --output json", - "file": "or-jobs-logs-cb030003-level-error-output-json.json" - }, - { - "match": "or jobs logs cb030003-e5f6-4071-9b03-000000000003 --level Error --output json", - "file": "or-jobs-logs-cb030003-level-error-output-json.json" - }, - { - "match": "or jobs logs \"cb030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-logs-cb030003-output-json.json" - }, - { - "match": "or jobs logs cb030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-logs-cb030003-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 9717163203..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ca030003-d4e5-4f60-8a03-000000000003", - "Name": "Data Integration", - "Path": "Data Integration", - "Description": "Production folder for CSV merge/integration processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-get-cb030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-get-cb030003-output-json.json deleted file mode 100644 index 9aa0c01dd9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-get-cb030003-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cb030003-e5f6-4071-9b03-000000000003", - "StartTime": "2026-06-13T11:11:08.210Z", - "EndTime": "2026-06-13T11:11:10.330Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc030003-f6a7-4182-9c03-000000000003", - "Info": "Append To CSV: The input DataTable has 2 column(s) but the existing file 'data\\orders.csv' has 3 column(s) (OrderId, Customer, Total). The rows cannot be appended because the column structure does not match.\r\n\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidOperationException: DataTable column count does not match the target CSV.", - "CreationTime": "2026-06-13T11:11:07.800Z", - "ReleaseName": "CsvMerger", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7013003, - "OrganizationUnitId": 7791618, - "FolderKey": "ca030003-d4e5-4f60-8a03-000000000003", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-13T11:11:10.330Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "cb030003-e5f6-4071-9b03-000000000003", - "Id": 713010303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-list-folder-key-ca030003-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-list-folder-key-ca030003-state-faulted-output-json.json deleted file mode 100644 index cebddc0e53..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-list-folder-key-ca030003-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "cb030003-e5f6-4071-9b03-000000000003", - "State": "Faulted", - "ProcessName": "CsvMerger", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-13T11:11:08.210Z", - "EndTime": "2026-06-13T11:11:10.330Z", - "CreationTime": "2026-06-13T11:11:07.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-logs-cb030003-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-logs-cb030003-level-error-output-json.json deleted file mode 100644 index b5d3238616..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-logs-cb030003-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-13T11:11:10.210Z", - "Message": "[Append To CSV] The input DataTable has 2 column(s) but the existing file 'data\\orders.csv' has 3 column(s) (OrderId, Customer, Total). The rows cannot be appended because the column structure does not match." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T11:11:10.270Z", - "Message": "Append To CSV: DataTable column count does not match the target CSV.\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-logs-cb030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-logs-cb030003-output-json.json deleted file mode 100644 index 6079e93a49..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/m/r/or-jobs-logs-cb030003-output-json.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-13T11:11:08.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-13T11:11:09.310Z", - "Message": "[Add Data Column - OrderId] Added column 'OrderId' to the DataTable" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-13T11:11:09.520Z", - "Message": "[Add Data Column - Customer] Added column 'Customer' to the DataTable" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T11:11:10.210Z", - "Message": "[Append To CSV] The input DataTable has 2 column(s) but the existing file 'data\\orders.csv' has 3 column(s) (OrderId, Customer, Total). The rows cannot be appended because the column structure does not match." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T11:11:10.270Z", - "Message": "Append To CSV: DataTable column count does not match the target CSV.\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T11:11:10.300Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..8bcec1db8d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ca030003-d4e5-4f60-8a03-000000000003", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cb030003-e5f6-4071-9b03-000000000003\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvMerger\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-13T11:11:08.210Z\",\r\n \"EndTime\": \"2026-06-13T11:11:10.330Z\",\r\n \"CreationTime\": \"2026-06-13T11:11:07.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cb030003-e5f6-4071-9b03-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T11:11:10.210Z\",\r\n \"Message\": \"[Append To CSV] The input DataTable has 2 column(s) but the existing file 'data\\\\orders.csv' has 3 column(s) (OrderId, Customer, Total). The rows cannot be appended because the column structure does not match.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T11:11:10.270Z\",\r\n \"Message\": \"Append To CSV: DataTable column count does not match the target CSV.\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cb030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cb030003-e5f6-4071-9b03-000000000003\",\r\n \"StartTime\": \"2026-06-13T11:11:08.210Z\",\r\n \"EndTime\": \"2026-06-13T11:11:10.330Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc030003-f6a7-4182-9c03-000000000003\",\r\n \"Info\": \"Append To CSV: The input DataTable has 2 column(s) but the existing file 'data\\\\orders.csv' has 3 column(s) (OrderId, Customer, Total). The rows cannot be appended because the column structure does not match.\\r\\n\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidOperationException: DataTable column count does not match the target CSV.\",\r\n \"CreationTime\": \"2026-06-13T11:11:07.800Z\",\r\n \"ReleaseName\": \"CsvMerger\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7013003,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ca030003-d4e5-4f60-8a03-000000000003\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-13T11:11:10.330Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"cb030003-e5f6-4071-9b03-000000000003\",\r\n \"Id\": 713010303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cb030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T11:11:08.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-13T11:11:09.310Z\",\r\n \"Message\": \"[Add Data Column - OrderId] Added column 'OrderId' to the DataTable\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-13T11:11:09.520Z\",\r\n \"Message\": \"[Add Data Column - Customer] Added column 'Customer' to the DataTable\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T11:11:10.210Z\",\r\n \"Message\": \"[Append To CSV] The input DataTable has 2 column(s) but the existing file 'data\\\\orders.csv' has 3 column(s) (OrderId, Customer, Total). The rows cannot be appended because the column structure does not match.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T11:11:10.270Z\",\r\n \"Message\": \"Append To CSV: DataTable column count does not match the target CSV.\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T11:11:10.300Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ca030003-d4e5-4f60-8a03-000000000003\",\r\n \"Name\": \"Data Integration\",\r\n \"Path\": \"Data Integration\",\r\n \"Description\": \"Production folder for CSV merge/integration processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/task.yaml index 0e47f48523..b15c4461ac 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-column-mismatch/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/README.md new file mode 100644 index 0000000000..a3d28bf721 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/README.md @@ -0,0 +1,42 @@ +# Append To CSV Failure - File Locked (Open in Excel) + +This scenario reproduces an `Append To CSV` failure caused by a **file lock**. +The target `daily.csv` is open in Microsoft Excel on the robot host, so the +append cannot acquire exclusive write access and faults with `The process cannot +access the file ... because it is being used by another process`. + +## What this scenario uncovers + +**Root Cause:** `daily.csv` is held by another process (open in Excel / another +session), so `Append To CSV` cannot write to it (`System.IO.IOException`). + +This maps to: +`references/activity-packages/csv-activities/playbooks/csv-file-locked-or-invalid-path.md` + +The discriminator vs the other CSV playbooks: the error is a **lock** +(`being used by another process`), not a `CsvHelper` `Method not found` and not a +`DataTable` shape problem. The user is framed as **off-host**, so the correct +agent behavior is to recommend closing/killing the Excel lock holder (and/or +serializing access) - not to run host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project that Reads a CSV then `Append To CSV` to `data\daily.csv` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `csv-file-locked-or-invalid-path.md` +- Agent identified the file lock (open in Excel / held by another process) and + recommended closing or killing the lock holder (e.g. Kill Process EXCEL before + the append) and/or serializing access, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/README.md deleted file mode 100644 index 85084b1c23..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Append To CSV Failure - File Locked (Open in Excel) - -This scenario reproduces an `Append To CSV` failure caused by a **file lock**. -The target `daily.csv` is open in Microsoft Excel on the robot host, so the -append cannot acquire exclusive write access and faults with `The process cannot -access the file ... because it is being used by another process`. - -## What this scenario uncovers - -**Root Cause:** `daily.csv` is held by another process (open in Excel / another -session), so `Append To CSV` cannot write to it (`System.IO.IOException`). - -This maps to: -`references/activity-packages/csv-activities/playbooks/csv-file-locked-or-invalid-path.md` - -The discriminator vs the other CSV playbooks: the error is a **lock** -(`being used by another process`), not a `CsvHelper` `Method not found` and not a -`DataTable` shape problem. The user is framed as **off-host**, so the correct -agent behavior is to recommend closing/killing the Excel lock holder (and/or -serializing access) - not to run host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project that Reads a CSV then `Append To CSV` to `data\daily.csv` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `csv-file-locked-or-invalid-path.md` -- Agent identified the file lock (open in Excel / held by another process) and - recommended closing or killing the lock holder (e.g. Kill Process EXCEL before - the append) and/or serializing access, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/manifest.json deleted file mode 100644 index 2af94b9cff..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ca020002-d4e5-4f60-8a02-000000000002\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ca020002-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ca020002-d4e5-4f60-8a02-000000000002 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ca020002-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"cb020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-get-cb020002-output-json.json" - }, - { - "match": "or jobs get cb020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-get-cb020002-output-json.json" - }, - { - "match": "or jobs logs \"cb020002-e5f6-4071-9b02-000000000002\" --level Error --output json", - "file": "or-jobs-logs-cb020002-level-error-output-json.json" - }, - { - "match": "or jobs logs cb020002-e5f6-4071-9b02-000000000002 --level Error --output json", - "file": "or-jobs-logs-cb020002-level-error-output-json.json" - }, - { - "match": "or jobs logs \"cb020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-logs-cb020002-output-json.json" - }, - { - "match": "or jobs logs cb020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-logs-cb020002-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 252d964fdf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ca020002-d4e5-4f60-8a02-000000000002", - "Name": "Daily Loads", - "Path": "Daily Loads", - "Description": "Production folder for daily CSV append processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-get-cb020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-get-cb020002-output-json.json deleted file mode 100644 index 760f8f9a91..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-get-cb020002-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cb020002-e5f6-4071-9b02-000000000002", - "StartTime": "2026-06-13T09:30:41.210Z", - "EndTime": "2026-06-13T09:30:43.470Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc020002-f6a7-4182-9c02-000000000002", - "Info": "Append To CSV: The process cannot access the file 'C:\\ProgramData\\UiPath\\Packages\\CsvDailyAppend\\data\\daily.csv' because it is being used by another process.\r\n\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.IOException: The process cannot access the file because it is being used by another process.", - "CreationTime": "2026-06-13T09:30:40.800Z", - "ReleaseName": "CsvDailyAppend", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7013002, - "OrganizationUnitId": 7791618, - "FolderKey": "ca020002-d4e5-4f60-8a02-000000000002", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-13T09:30:43.470Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "cb020002-e5f6-4071-9b02-000000000002", - "Id": 713010202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-list-folder-key-ca020002-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-list-folder-key-ca020002-state-faulted-output-json.json deleted file mode 100644 index 585c05af31..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-list-folder-key-ca020002-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "cb020002-e5f6-4071-9b02-000000000002", - "State": "Faulted", - "ProcessName": "CsvDailyAppend", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-13T09:30:41.210Z", - "EndTime": "2026-06-13T09:30:43.470Z", - "CreationTime": "2026-06-13T09:30:40.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-logs-cb020002-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-logs-cb020002-level-error-output-json.json deleted file mode 100644 index ce446fd767..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-logs-cb020002-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-13T09:30:43.360Z", - "Message": "[Append To CSV] The process cannot access the file 'C:\\ProgramData\\UiPath\\Packages\\CsvDailyAppend\\data\\daily.csv' because it is being used by another process." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T09:30:43.420Z", - "Message": "Append To CSV: The process cannot access the file because it is being used by another process.\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-logs-cb020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-logs-cb020002-output-json.json deleted file mode 100644 index 86477959dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/m/r/or-jobs-logs-cb020002-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-13T09:30:41.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-13T09:30:42.480Z", - "Message": "[Read CSV] Read 12 rows from data\\incoming.csv" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T09:30:43.360Z", - "Message": "[Append To CSV] The process cannot access the file 'C:\\ProgramData\\UiPath\\Packages\\CsvDailyAppend\\data\\daily.csv' because it is being used by another process." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T09:30:43.420Z", - "Message": "Append To CSV: The process cannot access the file because it is being used by another process.\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T09:30:43.450Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/uip-fixture.json new file mode 100644 index 0000000000..db0a5c18bb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ca020002-d4e5-4f60-8a02-000000000002", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cb020002-e5f6-4071-9b02-000000000002\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvDailyAppend\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-13T09:30:41.210Z\",\r\n \"EndTime\": \"2026-06-13T09:30:43.470Z\",\r\n \"CreationTime\": \"2026-06-13T09:30:40.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cb020002-e5f6-4071-9b02-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T09:30:43.360Z\",\r\n \"Message\": \"[Append To CSV] The process cannot access the file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvDailyAppend\\\\data\\\\daily.csv' because it is being used by another process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T09:30:43.420Z\",\r\n \"Message\": \"Append To CSV: The process cannot access the file because it is being used by another process.\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cb020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cb020002-e5f6-4071-9b02-000000000002\",\r\n \"StartTime\": \"2026-06-13T09:30:41.210Z\",\r\n \"EndTime\": \"2026-06-13T09:30:43.470Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc020002-f6a7-4182-9c02-000000000002\",\r\n \"Info\": \"Append To CSV: The process cannot access the file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvDailyAppend\\\\data\\\\daily.csv' because it is being used by another process.\\r\\n\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.IOException: The process cannot access the file because it is being used by another process.\",\r\n \"CreationTime\": \"2026-06-13T09:30:40.800Z\",\r\n \"ReleaseName\": \"CsvDailyAppend\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7013002,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ca020002-d4e5-4f60-8a02-000000000002\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-13T09:30:43.470Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"cb020002-e5f6-4071-9b02-000000000002\",\r\n \"Id\": 713010202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cb020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T09:30:41.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-13T09:30:42.480Z\",\r\n \"Message\": \"[Read CSV] Read 12 rows from data\\\\incoming.csv\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T09:30:43.360Z\",\r\n \"Message\": \"[Append To CSV] The process cannot access the file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvDailyAppend\\\\data\\\\daily.csv' because it is being used by another process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T09:30:43.420Z\",\r\n \"Message\": \"Append To CSV: The process cannot access the file because it is being used by another process.\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T09:30:43.450Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ca020002-d4e5-4f60-8a02-000000000002\",\r\n \"Name\": \"Daily Loads\",\r\n \"Path\": \"Daily Loads\",\r\n \"Description\": \"Production folder for daily CSV append processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/task.yaml index c1a52b7005..a1a02c5764 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-append-file-locked/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/README.md new file mode 100644 index 0000000000..8ef57f519f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/README.md @@ -0,0 +1,45 @@ +# Append To CSV Failure - CsvHelper Version Conflict (Method not found) + +This scenario reproduces an `Append To CSV` failure caused by a **CsvHelper +version conflict**. `CsvHelper.dll` is bundled in **both** +`UiPath.System.Activities` (which provides the CSV activities) and +`UiPath.Excel.Activities`. The project pins **mismatched versions** of the two, +so the CSV activity binds an incompatible `CsvHelper` and faults with +`Method not found: 'Void CsvHelper.CsvWriter..ctor(...)'`. + +## What this scenario uncovers + +**Root Cause:** `project.json` pins `UiPath.System.Activities [23.4.0]` and +`UiPath.Excel.Activities [2.24.4]` — versions that bundle incompatible +`CsvHelper` builds. `Append To CSV` is compiled against one `CsvHelper` API but +binds the other at runtime, raising `MissingMethodException`. + +This maps to: +`references/activity-packages/csv-activities/playbooks/csv-helper-method-not-found.md` + +The fix is a Studio/project dependency change (align both packages), so the +correct agent behavior is to identify the System↔Excel CsvHelper conflict and +recommend upgrading both — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project that Reads Range (Excel) then Append To CSV, with `project.json` pinning mismatched `UiPath.System.Activities` / `UiPath.Excel.Activities` versions | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `csv-helper-method-not-found.md` +- Agent identified the CsvHelper conflict between `UiPath.System.Activities` and + `UiPath.Excel.Activities` (mismatched versions bundling incompatible CsvHelper) + and recommended aligning/upgrading both packages, without fabricating host + actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/README.md deleted file mode 100644 index e06c481f61..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Append To CSV Failure - CsvHelper Version Conflict (Method not found) - -This scenario reproduces an `Append To CSV` failure caused by a **CsvHelper -version conflict**. `CsvHelper.dll` is bundled in **both** -`UiPath.System.Activities` (which provides the CSV activities) and -`UiPath.Excel.Activities`. The project pins **mismatched versions** of the two, -so the CSV activity binds an incompatible `CsvHelper` and faults with -`Method not found: 'Void CsvHelper.CsvWriter..ctor(...)'`. - -## What this scenario uncovers - -**Root Cause:** `project.json` pins `UiPath.System.Activities [23.4.0]` and -`UiPath.Excel.Activities [2.24.4]` — versions that bundle incompatible -`CsvHelper` builds. `Append To CSV` is compiled against one `CsvHelper` API but -binds the other at runtime, raising `MissingMethodException`. - -This maps to: -`references/activity-packages/csv-activities/playbooks/csv-helper-method-not-found.md` - -The fix is a Studio/project dependency change (align both packages), so the -correct agent behavior is to identify the System↔Excel CsvHelper conflict and -recommend upgrading both — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project that Reads Range (Excel) then Append To CSV, with `project.json` pinning mismatched `UiPath.System.Activities` / `UiPath.Excel.Activities` versions | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `csv-helper-method-not-found.md` -- Agent identified the CsvHelper conflict between `UiPath.System.Activities` and - `UiPath.Excel.Activities` (mismatched versions bundling incompatible CsvHelper) - and recommended aligning/upgrading both packages, without fabricating host - actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/manifest.json deleted file mode 100644 index 3664c059ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ca010001-d4e5-4f60-8a01-000000000001\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ca010001-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ca010001-d4e5-4f60-8a01-000000000001 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ca010001-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"cb010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-get-cb010001-output-json.json" - }, - { - "match": "or jobs get cb010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-get-cb010001-output-json.json" - }, - { - "match": "or jobs logs \"cb010001-e5f6-4071-9b01-000000000001\" --level Error --output json", - "file": "or-jobs-logs-cb010001-level-error-output-json.json" - }, - { - "match": "or jobs logs cb010001-e5f6-4071-9b01-000000000001 --level Error --output json", - "file": "or-jobs-logs-cb010001-level-error-output-json.json" - }, - { - "match": "or jobs logs \"cb010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-logs-cb010001-output-json.json" - }, - { - "match": "or jobs logs cb010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-logs-cb010001-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 3acd176dba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ca010001-d4e5-4f60-8a01-000000000001", - "Name": "Reporting", - "Path": "Reporting", - "Description": "Production folder for reporting/export processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-get-cb010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-get-cb010001-output-json.json deleted file mode 100644 index db5cf01b3e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-get-cb010001-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cb010001-e5f6-4071-9b01-000000000001", - "StartTime": "2026-06-13T08:02:14.210Z", - "EndTime": "2026-06-13T08:02:16.540Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc010001-f6a7-4182-9c01-000000000001", - "Info": "Append To CSV: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\r\n\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.MissingMethodException: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.", - "CreationTime": "2026-06-13T08:02:13.800Z", - "ReleaseName": "CsvExporter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7013001, - "OrganizationUnitId": 7791618, - "FolderKey": "ca010001-d4e5-4f60-8a01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-13T08:02:16.540Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "cb010001-e5f6-4071-9b01-000000000001", - "Id": 713010101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-list-folder-key-ca010001-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-list-folder-key-ca010001-state-faulted-output-json.json deleted file mode 100644 index 3de35b489c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-list-folder-key-ca010001-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "cb010001-e5f6-4071-9b01-000000000001", - "State": "Faulted", - "ProcessName": "CsvExporter", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-13T08:02:14.210Z", - "EndTime": "2026-06-13T08:02:16.540Z", - "CreationTime": "2026-06-13T08:02:13.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-logs-cb010001-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-logs-cb010001-level-error-output-json.json deleted file mode 100644 index b3f5f3feaa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-logs-cb010001-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-13T08:02:16.420Z", - "Message": "[Append To CSV] Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T08:02:16.470Z", - "Message": "Append To CSV: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-logs-cb010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-logs-cb010001-output-json.json deleted file mode 100644 index 833ff274b7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/m/r/or-jobs-logs-cb010001-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-13T08:02:14.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-13T08:02:15.560Z", - "Message": "[Read Range] Read 42 rows from data\\source.xlsx (Sheet1)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T08:02:16.420Z", - "Message": "[Append To CSV] Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T08:02:16.470Z", - "Message": "Append To CSV: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\r\nin Main.xaml\r\n at AppendToCsvFile \"Append To CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T08:02:16.520Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..4826903367 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ca010001-d4e5-4f60-8a01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cb010001-e5f6-4071-9b01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvExporter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-13T08:02:14.210Z\",\r\n \"EndTime\": \"2026-06-13T08:02:16.540Z\",\r\n \"CreationTime\": \"2026-06-13T08:02:13.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cb010001-e5f6-4071-9b01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T08:02:16.420Z\",\r\n \"Message\": \"[Append To CSV] Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T08:02:16.470Z\",\r\n \"Message\": \"Append To CSV: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cb010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cb010001-e5f6-4071-9b01-000000000001\",\r\n \"StartTime\": \"2026-06-13T08:02:14.210Z\",\r\n \"EndTime\": \"2026-06-13T08:02:16.540Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc010001-f6a7-4182-9c01-000000000001\",\r\n \"Info\": \"Append To CSV: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\\r\\n\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.MissingMethodException: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\",\r\n \"CreationTime\": \"2026-06-13T08:02:13.800Z\",\r\n \"ReleaseName\": \"CsvExporter\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7013001,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ca010001-d4e5-4f60-8a01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-13T08:02:16.540Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"cb010001-e5f6-4071-9b01-000000000001\",\r\n \"Id\": 713010101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cb010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T08:02:14.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-13T08:02:15.560Z\",\r\n \"Message\": \"[Read Range] Read 42 rows from data\\\\source.xlsx (Sheet1)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T08:02:16.420Z\",\r\n \"Message\": \"[Append To CSV] Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T08:02:16.470Z\",\r\n \"Message\": \"Append To CSV: Method not found: 'Void CsvHelper.CsvWriter..ctor(System.IO.TextWriter, CsvHelper.Configuration.CsvConfiguration)'.\\r\\nin Main.xaml\\r\\n at AppendToCsvFile \\\"Append To CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T08:02:16.520Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ca010001-d4e5-4f60-8a01-000000000001\",\r\n \"Name\": \"Reporting\",\r\n \"Path\": \"Reporting\",\r\n \"Description\": \"Production folder for reporting/export processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/task.yaml index 4e17296324..e6e7cea027 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-csvhelper-method-not-found/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/README.md new file mode 100644 index 0000000000..d78a007b06 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/README.md @@ -0,0 +1,43 @@ +# Read CSV Failure - "Could not find file" (Async Download Timing) + +This scenario reproduces a `Read CSV` failure where the CSV is produced by an +**upstream download immediately before the read**, and `Read CSV` runs before the +file is finished writing to disk — so it faults with `Could not find file +'...\data\daily-export.csv'` (`FileNotFoundException`). The file is present when +checked afterward, which points at a timing race rather than a permanently wrong +path. + +## What this scenario uncovers + +**Root Cause:** A download/export step runs just before `Read CSV`; at read time +the file isn't on disk yet (async write not complete), so the read fails +intermittently. (The playbook also covers a path variable wrapped in quotes and +relative-path drift.) + +This maps to: +`references/activity-packages/csv-activities/playbooks/read-csv-file-not-found.md` + +The fix is a workflow change (wait for the file: File Exists retry / Delay; +verify the path), not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project that reads `data\daily-export.csv` with `Read CSV`; the file is produced by an upstream download (not committed, mirroring "not on disk yet") | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the async-download timing + race (file not written yet at read time; intermittent) and recommends guarding + the read with a File Exists retry / Delay (and verifying the path / unquoted + path variable), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/README.md deleted file mode 100644 index 71ebc34f04..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Read CSV Failure - "Could not find file" (Async Download Timing) - -This scenario reproduces a `Read CSV` failure where the CSV is produced by an -**upstream download immediately before the read**, and `Read CSV` runs before the -file is finished writing to disk — so it faults with `Could not find file -'...\data\daily-export.csv'` (`FileNotFoundException`). The file is present when -checked afterward, which points at a timing race rather than a permanently wrong -path. - -## What this scenario uncovers - -**Root Cause:** A download/export step runs just before `Read CSV`; at read time -the file isn't on disk yet (async write not complete), so the read fails -intermittently. (The playbook also covers a path variable wrapped in quotes and -relative-path drift.) - -This maps to: -`references/activity-packages/csv-activities/playbooks/read-csv-file-not-found.md` - -The fix is a workflow change (wait for the file: File Exists retry / Delay; -verify the path), not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project that reads `data\daily-export.csv` with `Read CSV`; the file is produced by an upstream download (not committed, mirroring "not on disk yet") | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; logs show the download step immediately before the failing read | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the async-download timing - race (file not written yet at read time; intermittent) and recommends guarding - the read with a File Exists retry / Delay (and verifying the path / unquoted - path variable), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/manifest.json deleted file mode 100644 index e5582c1fde..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"da030003-d4e5-4f60-8a03-000000000003\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-da030003-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key da030003-d4e5-4f60-8a03-000000000003 --state Faulted --output json", - "file": "or-jobs-list-folder-key-da030003-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"db030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-get-db030003-output-json.json" - }, - { - "match": "or jobs get db030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-get-db030003-output-json.json" - }, - { - "match": "or jobs logs \"db030003-e5f6-4071-9b03-000000000003\" --level Error --output json", - "file": "or-jobs-logs-db030003-level-error-output-json.json" - }, - { - "match": "or jobs logs db030003-e5f6-4071-9b03-000000000003 --level Error --output json", - "file": "or-jobs-logs-db030003-level-error-output-json.json" - }, - { - "match": "or jobs logs \"db030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-logs-db030003-output-json.json" - }, - { - "match": "or jobs logs db030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-logs-db030003-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 4b1ecb948c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "da030003-d4e5-4f60-8a03-000000000003", - "Name": "Downloads", - "Path": "Downloads", - "Description": "Production folder for download-and-read processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-get-db030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-get-db030003-output-json.json deleted file mode 100644 index 376cf851d0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-get-db030003-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "db030003-e5f6-4071-9b03-000000000003", - "StartTime": "2026-06-14T13:55:30.210Z", - "EndTime": "2026-06-14T13:55:32.180Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "dc030003-f6a7-4182-9c03-000000000003", - "Info": "Read CSV: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\CsvFetcher\\data\\daily-export.csv'.\r\n\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileNotFoundException: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\CsvFetcher\\data\\daily-export.csv'.", - "CreationTime": "2026-06-14T13:55:29.800Z", - "ReleaseName": "CsvFetcher", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7014003, - "OrganizationUnitId": 7791618, - "FolderKey": "da030003-d4e5-4f60-8a03-000000000003", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-14T13:55:32.180Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "db030003-e5f6-4071-9b03-000000000003", - "Id": 714010303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-list-folder-key-da030003-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-list-folder-key-da030003-state-faulted-output-json.json deleted file mode 100644 index 976676238f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-list-folder-key-da030003-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "db030003-e5f6-4071-9b03-000000000003", - "State": "Faulted", - "ProcessName": "CsvFetcher", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-14T13:55:30.210Z", - "EndTime": "2026-06-14T13:55:32.180Z", - "CreationTime": "2026-06-14T13:55:29.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-logs-db030003-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-logs-db030003-level-error-output-json.json deleted file mode 100644 index 6c2ae3acd7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-logs-db030003-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-14T13:55:32.020Z", - "Message": "[Read CSV] Could not find file 'C:\\ProgramData\\UiPath\\Packages\\CsvFetcher\\data\\daily-export.csv'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T13:55:32.090Z", - "Message": "Read CSV: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\CsvFetcher\\data\\daily-export.csv'.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-logs-db030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-logs-db030003-output-json.json deleted file mode 100644 index c7994054c4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/m/r/or-jobs-logs-db030003-output-json.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-14T13:55:30.240Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T13:55:31.060Z", - "Message": "[Download Daily Export] Requested daily-export.csv from the reporting API" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T13:55:31.640Z", - "Message": "[Read CSV] Reading data\\daily-export.csv (Delimiter=Comma, AddHeaders=True)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T13:55:32.020Z", - "Message": "[Read CSV] Could not find file 'C:\\ProgramData\\UiPath\\Packages\\CsvFetcher\\data\\daily-export.csv'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T13:55:32.090Z", - "Message": "Read CSV: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\CsvFetcher\\data\\daily-export.csv'.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T13:55:32.150Z", - "Message": "Execution ended in: 00:00:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..3e10c3cc42 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "da030003-d4e5-4f60-8a03-000000000003", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"db030003-e5f6-4071-9b03-000000000003\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvFetcher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-14T13:55:30.210Z\",\r\n \"EndTime\": \"2026-06-14T13:55:32.180Z\",\r\n \"CreationTime\": \"2026-06-14T13:55:29.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "db030003-e5f6-4071-9b03-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T13:55:32.020Z\",\r\n \"Message\": \"[Read CSV] Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvFetcher\\\\data\\\\daily-export.csv'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T13:55:32.090Z\",\r\n \"Message\": \"Read CSV: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvFetcher\\\\data\\\\daily-export.csv'.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "db030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"db030003-e5f6-4071-9b03-000000000003\",\r\n \"StartTime\": \"2026-06-14T13:55:30.210Z\",\r\n \"EndTime\": \"2026-06-14T13:55:32.180Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"dc030003-f6a7-4182-9c03-000000000003\",\r\n \"Info\": \"Read CSV: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvFetcher\\\\data\\\\daily-export.csv'.\\r\\n\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvFetcher\\\\data\\\\daily-export.csv'.\",\r\n \"CreationTime\": \"2026-06-14T13:55:29.800Z\",\r\n \"ReleaseName\": \"CsvFetcher\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7014003,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"da030003-d4e5-4f60-8a03-000000000003\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-14T13:55:32.180Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"db030003-e5f6-4071-9b03-000000000003\",\r\n \"Id\": 714010303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "db030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T13:55:30.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T13:55:31.060Z\",\r\n \"Message\": \"[Download Daily Export] Requested daily-export.csv from the reporting API\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T13:55:31.640Z\",\r\n \"Message\": \"[Read CSV] Reading data\\\\daily-export.csv (Delimiter=Comma, AddHeaders=True)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T13:55:32.020Z\",\r\n \"Message\": \"[Read CSV] Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvFetcher\\\\data\\\\daily-export.csv'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T13:55:32.090Z\",\r\n \"Message\": \"Read CSV: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvFetcher\\\\data\\\\daily-export.csv'.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T13:55:32.150Z\",\r\n \"Message\": \"Execution ended in: 00:00:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"da030003-d4e5-4f60-8a03-000000000003\",\r\n \"Name\": \"Downloads\",\r\n \"Path\": \"Downloads\",\r\n \"Description\": \"Production folder for download-and-read processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/task.yaml index 5521b66799..c60247a878 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-file-not-found/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/README.md new file mode 100644 index 0000000000..3c4e761537 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/README.md @@ -0,0 +1,40 @@ +# Read CSV Failure - "The CSV file format for [Path] is invalid" + +This scenario reproduces a `Read CSV` failure caused by **leading blank lines** +at the top of the file while `AddHeaders` is enabled. `data/feed.csv` starts with +two empty rows before the `Id,Sku,Qty` header, so `Read CSV` cannot resolve the +tabular structure and faults with `The CSV file format for 'data\feed.csv' is +invalid`. + +## What this scenario uncovers + +**Root Cause:** Blank initial lines shift header detection — `Read CSV` (with +`AddHeaders=True`) expects the header at the top, but finds empty rows first. + +This maps to: +`references/activity-packages/csv-activities/playbooks/read-csv-invalid-format.md` + +The fix is a file/property change (strip the leading blank lines; align +`Has headers`/`AddHeaders` with the file) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Read CSV` (`AddHeaders=True`) and `data/feed.csv` that has two blank lines before the header | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the leading-blank-lines + (and/or `Has headers` mismatch) cause and recommends stripping the leading + blank lines (and aligning `Has headers`/`AddHeaders`), without fabricating host + actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/README.md deleted file mode 100644 index bcefbb51ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Read CSV Failure - "The CSV file format for [Path] is invalid" - -This scenario reproduces a `Read CSV` failure caused by **leading blank lines** -at the top of the file while `AddHeaders` is enabled. `data/feed.csv` starts with -two empty rows before the `Id,Sku,Qty` header, so `Read CSV` cannot resolve the -tabular structure and faults with `The CSV file format for 'data\feed.csv' is -invalid`. - -## What this scenario uncovers - -**Root Cause:** Blank initial lines shift header detection — `Read CSV` (with -`AddHeaders=True`) expects the header at the top, but finds empty rows first. - -This maps to: -`references/activity-packages/csv-activities/playbooks/read-csv-invalid-format.md` - -The fix is a file/property change (strip the leading blank lines; align -`Has headers`/`AddHeaders` with the file) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Read CSV` (`AddHeaders=True`) and `data/feed.csv` that has two blank lines before the header | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the leading-blank-lines - (and/or `Has headers` mismatch) cause and recommends stripping the leading - blank lines (and aligning `Has headers`/`AddHeaders`), without fabricating host - actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/manifest.json deleted file mode 100644 index 4bb7ee77c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"da020002-d4e5-4f60-8a02-000000000002\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-da020002-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key da020002-d4e5-4f60-8a02-000000000002 --state Faulted --output json", - "file": "or-jobs-list-folder-key-da020002-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"db020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-get-db020002-output-json.json" - }, - { - "match": "or jobs get db020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-get-db020002-output-json.json" - }, - { - "match": "or jobs logs \"db020002-e5f6-4071-9b02-000000000002\" --level Error --output json", - "file": "or-jobs-logs-db020002-level-error-output-json.json" - }, - { - "match": "or jobs logs db020002-e5f6-4071-9b02-000000000002 --level Error --output json", - "file": "or-jobs-logs-db020002-level-error-output-json.json" - }, - { - "match": "or jobs logs \"db020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-logs-db020002-output-json.json" - }, - { - "match": "or jobs logs db020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-logs-db020002-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 2125c9b3a1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "da020002-d4e5-4f60-8a02-000000000002", - "Name": "Feeds", - "Path": "Feeds", - "Description": "Production folder for feed-loading processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-get-db020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-get-db020002-output-json.json deleted file mode 100644 index 1e3aa8a73a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-get-db020002-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "db020002-e5f6-4071-9b02-000000000002", - "StartTime": "2026-06-14T10:41:18.210Z", - "EndTime": "2026-06-14T10:41:20.300Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "dc020002-f6a7-4182-9c02-000000000002", - "Info": "Read CSV: The CSV file format for 'data\\feed.csv' is invalid.\r\n\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.InvalidDataException: The CSV file format for 'data\\feed.csv' is invalid.", - "CreationTime": "2026-06-14T10:41:17.800Z", - "ReleaseName": "FeedLoader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7014002, - "OrganizationUnitId": 7791618, - "FolderKey": "da020002-d4e5-4f60-8a02-000000000002", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-14T10:41:20.300Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "db020002-e5f6-4071-9b02-000000000002", - "Id": 714010202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-list-folder-key-da020002-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-list-folder-key-da020002-state-faulted-output-json.json deleted file mode 100644 index a43f6b9986..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-list-folder-key-da020002-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "db020002-e5f6-4071-9b02-000000000002", - "State": "Faulted", - "ProcessName": "FeedLoader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-14T10:41:18.210Z", - "EndTime": "2026-06-14T10:41:20.300Z", - "CreationTime": "2026-06-14T10:41:17.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-logs-db020002-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-logs-db020002-level-error-output-json.json deleted file mode 100644 index 8d59c8d92c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-logs-db020002-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-14T10:41:20.150Z", - "Message": "[Read CSV] The CSV file format for 'data\\feed.csv' is invalid." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T10:41:20.220Z", - "Message": "Read CSV: The CSV file format for 'data\\feed.csv' is invalid.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-logs-db020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-logs-db020002-output-json.json deleted file mode 100644 index 4bca744afe..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/m/r/or-jobs-logs-db020002-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-14T10:41:18.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T10:41:19.480Z", - "Message": "[Read CSV] Reading data\\feed.csv (Delimiter=Comma, AddHeaders=True)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T10:41:20.150Z", - "Message": "[Read CSV] The CSV file format for 'data\\feed.csv' is invalid." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T10:41:20.220Z", - "Message": "Read CSV: The CSV file format for 'data\\feed.csv' is invalid.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T10:41:20.270Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/uip-fixture.json new file mode 100644 index 0000000000..2dd7dc9abb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "da020002-d4e5-4f60-8a02-000000000002", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"db020002-e5f6-4071-9b02-000000000002\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"FeedLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-14T10:41:18.210Z\",\r\n \"EndTime\": \"2026-06-14T10:41:20.300Z\",\r\n \"CreationTime\": \"2026-06-14T10:41:17.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "db020002-e5f6-4071-9b02-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T10:41:20.150Z\",\r\n \"Message\": \"[Read CSV] The CSV file format for 'data\\\\feed.csv' is invalid.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T10:41:20.220Z\",\r\n \"Message\": \"Read CSV: The CSV file format for 'data\\\\feed.csv' is invalid.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "db020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"db020002-e5f6-4071-9b02-000000000002\",\r\n \"StartTime\": \"2026-06-14T10:41:18.210Z\",\r\n \"EndTime\": \"2026-06-14T10:41:20.300Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"dc020002-f6a7-4182-9c02-000000000002\",\r\n \"Info\": \"Read CSV: The CSV file format for 'data\\\\feed.csv' is invalid.\\r\\n\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.InvalidDataException: The CSV file format for 'data\\\\feed.csv' is invalid.\",\r\n \"CreationTime\": \"2026-06-14T10:41:17.800Z\",\r\n \"ReleaseName\": \"FeedLoader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7014002,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"da020002-d4e5-4f60-8a02-000000000002\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-14T10:41:20.300Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"db020002-e5f6-4071-9b02-000000000002\",\r\n \"Id\": 714010202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "db020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T10:41:18.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T10:41:19.480Z\",\r\n \"Message\": \"[Read CSV] Reading data\\\\feed.csv (Delimiter=Comma, AddHeaders=True)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T10:41:20.150Z\",\r\n \"Message\": \"[Read CSV] The CSV file format for 'data\\\\feed.csv' is invalid.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T10:41:20.220Z\",\r\n \"Message\": \"Read CSV: The CSV file format for 'data\\\\feed.csv' is invalid.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T10:41:20.270Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"da020002-d4e5-4f60-8a02-000000000002\",\r\n \"Name\": \"Feeds\",\r\n \"Path\": \"Feeds\",\r\n \"Description\": \"Production folder for feed-loading processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/task.yaml index 5529ef3654..41fb50d7ce 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-invalid-format/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/README.md new file mode 100644 index 0000000000..049893daba --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/README.md @@ -0,0 +1,43 @@ +# Read CSV Failure - "Line X contains more values than the header line" + +This scenario reproduces a `Read CSV` parse failure. `data/contacts.csv` has a +3-column header (`Id,Name,Amount`) but a data row's `Amount` value is `1,250` — +an **unquoted comma inside a field**. With `Delimiter=Comma` and +`IgnoreQuotes=True`, that row parses into 4 fields, more than the 3-column +header, so `Read CSV` faults with `Line 4 contains more values than the header +line`. + +## What this scenario uncovers + +**Root Cause:** An unquoted comma in a data field (or, equivalently, a delimiter +that doesn't match the file) makes a row parse into more fields than the header. +`IgnoreQuotes=True` means quote characters aren't honored, so even a quoted +`"1,250"` would split. + +This maps to: +`references/activity-packages/csv-activities/playbooks/read-csv-more-values-than-header.md` + +The fix is a workflow/property/data change (correct `Delimiter`, honor quotes, +or Read Text File → Generate Data Table) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Read CSV` (`Delimiter=Comma`, `IgnoreQuotes=True`) and `data/contacts.csv` whose row 4 has an unquoted comma (`1,250`) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the unquoted-comma / + delimiter-quoting parse cause and recommends the corresponding fix (correct + `Delimiter`, honor quotes instead of `IgnoreQuotes`, quote the field, or Read + Text File → Generate Data Table), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/README.md deleted file mode 100644 index 1750ec933e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Read CSV Failure - "Line X contains more values than the header line" - -This scenario reproduces a `Read CSV` parse failure. `data/contacts.csv` has a -3-column header (`Id,Name,Amount`) but a data row's `Amount` value is `1,250` — -an **unquoted comma inside a field**. With `Delimiter=Comma` and -`IgnoreQuotes=True`, that row parses into 4 fields, more than the 3-column -header, so `Read CSV` faults with `Line 4 contains more values than the header -line`. - -## What this scenario uncovers - -**Root Cause:** An unquoted comma in a data field (or, equivalently, a delimiter -that doesn't match the file) makes a row parse into more fields than the header. -`IgnoreQuotes=True` means quote characters aren't honored, so even a quoted -`"1,250"` would split. - -This maps to: -`references/activity-packages/csv-activities/playbooks/read-csv-more-values-than-header.md` - -The fix is a workflow/property/data change (correct `Delimiter`, honor quotes, -or Read Text File → Generate Data Table) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Read CSV` (`Delimiter=Comma`, `IgnoreQuotes=True`) and `data/contacts.csv` whose row 4 has an unquoted comma (`1,250`) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the unquoted-comma / - delimiter-quoting parse cause and recommends the corresponding fix (correct - `Delimiter`, honor quotes instead of `IgnoreQuotes`, quote the field, or Read - Text File → Generate Data Table), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/manifest.json deleted file mode 100644 index bba45cb427..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"da010001-d4e5-4f60-8a01-000000000001\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-da010001-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key da010001-d4e5-4f60-8a01-000000000001 --state Faulted --output json", - "file": "or-jobs-list-folder-key-da010001-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"db010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-get-db010001-output-json.json" - }, - { - "match": "or jobs get db010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-get-db010001-output-json.json" - }, - { - "match": "or jobs logs \"db010001-e5f6-4071-9b01-000000000001\" --level Error --output json", - "file": "or-jobs-logs-db010001-level-error-output-json.json" - }, - { - "match": "or jobs logs db010001-e5f6-4071-9b01-000000000001 --level Error --output json", - "file": "or-jobs-logs-db010001-level-error-output-json.json" - }, - { - "match": "or jobs logs \"db010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-logs-db010001-output-json.json" - }, - { - "match": "or jobs logs db010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-logs-db010001-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index b362776a21..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "da010001-d4e5-4f60-8a01-000000000001", - "Name": "Imports", - "Path": "Imports", - "Description": "Production folder for CSV ingestion processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-get-db010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-get-db010001-output-json.json deleted file mode 100644 index 861c2e405a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-get-db010001-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "db010001-e5f6-4071-9b01-000000000001", - "StartTime": "2026-06-14T08:12:03.210Z", - "EndTime": "2026-06-14T08:12:05.330Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "dc010001-f6a7-4182-9c01-000000000001", - "Info": "Read CSV: Line 4 contains more values than the header line.\r\n\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nCsvHelper.MissingFieldException: Line 4 contains more values than the header line.", - "CreationTime": "2026-06-14T08:12:02.800Z", - "ReleaseName": "CsvIngest", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7014001, - "OrganizationUnitId": 7791618, - "FolderKey": "da010001-d4e5-4f60-8a01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-14T08:12:05.330Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "db010001-e5f6-4071-9b01-000000000001", - "Id": 714010101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-list-folder-key-da010001-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-list-folder-key-da010001-state-faulted-output-json.json deleted file mode 100644 index 89c3462d53..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-list-folder-key-da010001-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "db010001-e5f6-4071-9b01-000000000001", - "State": "Faulted", - "ProcessName": "CsvIngest", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-14T08:12:03.210Z", - "EndTime": "2026-06-14T08:12:05.330Z", - "CreationTime": "2026-06-14T08:12:02.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-logs-db010001-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-logs-db010001-level-error-output-json.json deleted file mode 100644 index 64051fb5c8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-logs-db010001-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-14T08:12:05.210Z", - "Message": "[Read CSV] Line 4 contains more values than the header line." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T08:12:05.270Z", - "Message": "Read CSV: Line 4 contains more values than the header line.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-logs-db010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-logs-db010001-output-json.json deleted file mode 100644 index 0672482a38..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/m/r/or-jobs-logs-db010001-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-14T08:12:03.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T08:12:04.510Z", - "Message": "[Read CSV] Reading data\\contacts.csv (Delimiter=Comma, AddHeaders=True, IgnoreQuotes=True)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T08:12:05.210Z", - "Message": "[Read CSV] Line 4 contains more values than the header line." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T08:12:05.270Z", - "Message": "Read CSV: Line 4 contains more values than the header line.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T08:12:05.300Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/uip-fixture.json new file mode 100644 index 0000000000..032fefe5cc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "da010001-d4e5-4f60-8a01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"db010001-e5f6-4071-9b01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvIngest\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-14T08:12:03.210Z\",\r\n \"EndTime\": \"2026-06-14T08:12:05.330Z\",\r\n \"CreationTime\": \"2026-06-14T08:12:02.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "db010001-e5f6-4071-9b01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T08:12:05.210Z\",\r\n \"Message\": \"[Read CSV] Line 4 contains more values than the header line.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T08:12:05.270Z\",\r\n \"Message\": \"Read CSV: Line 4 contains more values than the header line.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "db010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"db010001-e5f6-4071-9b01-000000000001\",\r\n \"StartTime\": \"2026-06-14T08:12:03.210Z\",\r\n \"EndTime\": \"2026-06-14T08:12:05.330Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"dc010001-f6a7-4182-9c01-000000000001\",\r\n \"Info\": \"Read CSV: Line 4 contains more values than the header line.\\r\\n\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nCsvHelper.MissingFieldException: Line 4 contains more values than the header line.\",\r\n \"CreationTime\": \"2026-06-14T08:12:02.800Z\",\r\n \"ReleaseName\": \"CsvIngest\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7014001,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"da010001-d4e5-4f60-8a01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-14T08:12:05.330Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"db010001-e5f6-4071-9b01-000000000001\",\r\n \"Id\": 714010101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "db010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T08:12:03.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T08:12:04.510Z\",\r\n \"Message\": \"[Read CSV] Reading data\\\\contacts.csv (Delimiter=Comma, AddHeaders=True, IgnoreQuotes=True)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T08:12:05.210Z\",\r\n \"Message\": \"[Read CSV] Line 4 contains more values than the header line.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T08:12:05.270Z\",\r\n \"Message\": \"Read CSV: Line 4 contains more values than the header line.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T08:12:05.300Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"da010001-d4e5-4f60-8a01-000000000001\",\r\n \"Name\": \"Imports\",\r\n \"Path\": \"Imports\",\r\n \"Description\": \"Production folder for CSV ingestion processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/task.yaml index 262862b3e3..86c42c7886 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-read-more-values-than-header/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/README.md new file mode 100644 index 0000000000..24b24310d2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/README.md @@ -0,0 +1,41 @@ +# Write CSV Failure - "Access to the path is denied" + +This scenario reproduces a `Write CSV` failure with `Access to the path +'...published.csv' is denied` (`UnauthorizedAccessException`). The output file is +read-only, open in Microsoft Excel, or in a folder the robot user cannot write +to. + +## What this scenario uncovers + +**Root Cause:** The robot cannot write the output file — permissions / read-only +attribute / an open Excel handle. + +This maps to: +`references/activity-packages/csv-activities/playbooks/write-csv-access-denied.md` + +The discriminator vs the file-lock playbook: this is `Access ... is denied` +(permissions/read-only/open-in-Excel), **not** `being used by another process`. +The fix is closing Excel / clearing read-only / granting write — not a host +command run by the agent. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project that reads a staging CSV and `Write CSV` to `data\published.csv` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the access-denied cause + (permissions / read-only / open in Excel) and recommends closing/Kill Process + EXCEL, clearing read-only, or granting the robot Read/Write (or writing to a + writable path), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/README.md deleted file mode 100644 index 4d8c66fc24..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# Write CSV Failure - "Access to the path is denied" - -This scenario reproduces a `Write CSV` failure with `Access to the path -'...published.csv' is denied` (`UnauthorizedAccessException`). The output file is -read-only, open in Microsoft Excel, or in a folder the robot user cannot write -to. - -## What this scenario uncovers - -**Root Cause:** The robot cannot write the output file — permissions / read-only -attribute / an open Excel handle. - -This maps to: -`references/activity-packages/csv-activities/playbooks/write-csv-access-denied.md` - -The discriminator vs the file-lock playbook: this is `Access ... is denied` -(permissions/read-only/open-in-Excel), **not** `being used by another process`. -The fix is closing Excel / clearing read-only / granting write — not a host -command run by the agent. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project that reads a staging CSV and `Write CSV` to `data\published.csv` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the access-denied cause - (permissions / read-only / open in Excel) and recommends closing/Kill Process - EXCEL, clearing read-only, or granting the robot Read/Write (or writing to a - writable path), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/manifest.json deleted file mode 100644 index c4389cc32d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ea030003-d4e5-4f60-8a03-000000000003\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea030003-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ea030003-d4e5-4f60-8a03-000000000003 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea030003-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"eb030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-get-eb030003-output-json.json" - }, - { - "match": "or jobs get eb030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-get-eb030003-output-json.json" - }, - { - "match": "or jobs logs \"eb030003-e5f6-4071-9b03-000000000003\" --level Error --output json", - "file": "or-jobs-logs-eb030003-level-error-output-json.json" - }, - { - "match": "or jobs logs eb030003-e5f6-4071-9b03-000000000003 --level Error --output json", - "file": "or-jobs-logs-eb030003-level-error-output-json.json" - }, - { - "match": "or jobs logs \"eb030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-logs-eb030003-output-json.json" - }, - { - "match": "or jobs logs eb030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-logs-eb030003-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 2247d8c824..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ea030003-d4e5-4f60-8a03-000000000003", - "Name": "Outputs", - "Path": "Outputs", - "Description": "Production folder for CSV publishing processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-get-eb030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-get-eb030003-output-json.json deleted file mode 100644 index efc0122b77..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-get-eb030003-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "eb030003-e5f6-4071-9b03-000000000003", - "StartTime": "2026-06-15T13:07:55.140Z", - "EndTime": "2026-06-15T13:07:57.060Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ec030003-f6a7-4182-9c03-000000000003", - "Info": "Write CSV: Access to the path 'C:\\ProgramData\\UiPath\\Packages\\CsvPublish\\data\\published.csv' is denied.\r\n\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.UnauthorizedAccessException: Access to the path 'C:\\ProgramData\\UiPath\\Packages\\CsvPublish\\data\\published.csv' is denied.", - "CreationTime": "2026-06-15T13:07:54.700Z", - "ReleaseName": "CsvPublish", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7015003, - "OrganizationUnitId": 7791618, - "FolderKey": "ea030003-d4e5-4f60-8a03-000000000003", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T13:07:57.060Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "eb030003-e5f6-4071-9b03-000000000003", - "Id": 715010303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-list-folder-key-ea030003-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-list-folder-key-ea030003-state-faulted-output-json.json deleted file mode 100644 index eaa8203826..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-list-folder-key-ea030003-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "eb030003-e5f6-4071-9b03-000000000003", - "State": "Faulted", - "ProcessName": "CsvPublish", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T13:07:55.140Z", - "EndTime": "2026-06-15T13:07:57.060Z", - "CreationTime": "2026-06-15T13:07:54.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-logs-eb030003-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-logs-eb030003-level-error-output-json.json deleted file mode 100644 index ae17032315..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-logs-eb030003-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T13:07:56.910Z", - "Message": "[Write CSV] Access to the path 'C:\\ProgramData\\UiPath\\Packages\\CsvPublish\\data\\published.csv' is denied." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T13:07:56.980Z", - "Message": "Write CSV: Access to the path 'C:\\ProgramData\\UiPath\\Packages\\CsvPublish\\data\\published.csv' is denied.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-logs-eb030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-logs-eb030003-output-json.json deleted file mode 100644 index 7c8b503e21..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/m/r/or-jobs-logs-eb030003-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T13:07:55.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T13:07:56.110Z", - "Message": "[Read CSV] Read 30 rows from data\\staging.csv" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T13:07:56.910Z", - "Message": "[Write CSV] Access to the path 'C:\\ProgramData\\UiPath\\Packages\\CsvPublish\\data\\published.csv' is denied." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T13:07:56.980Z", - "Message": "Write CSV: Access to the path 'C:\\ProgramData\\UiPath\\Packages\\CsvPublish\\data\\published.csv' is denied.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T13:07:57.030Z", - "Message": "Execution ended in: 00:00:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/uip-fixture.json new file mode 100644 index 0000000000..8bd7a7bd9f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ea030003-d4e5-4f60-8a03-000000000003", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"eb030003-e5f6-4071-9b03-000000000003\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvPublish\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T13:07:55.140Z\",\r\n \"EndTime\": \"2026-06-15T13:07:57.060Z\",\r\n \"CreationTime\": \"2026-06-15T13:07:54.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb030003-e5f6-4071-9b03-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T13:07:56.910Z\",\r\n \"Message\": \"[Write CSV] Access to the path 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvPublish\\\\data\\\\published.csv' is denied.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T13:07:56.980Z\",\r\n \"Message\": \"Write CSV: Access to the path 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvPublish\\\\data\\\\published.csv' is denied.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "eb030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"eb030003-e5f6-4071-9b03-000000000003\",\r\n \"StartTime\": \"2026-06-15T13:07:55.140Z\",\r\n \"EndTime\": \"2026-06-15T13:07:57.060Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ec030003-f6a7-4182-9c03-000000000003\",\r\n \"Info\": \"Write CSV: Access to the path 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvPublish\\\\data\\\\published.csv' is denied.\\r\\n\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.UnauthorizedAccessException: Access to the path 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvPublish\\\\data\\\\published.csv' is denied.\",\r\n \"CreationTime\": \"2026-06-15T13:07:54.700Z\",\r\n \"ReleaseName\": \"CsvPublish\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7015003,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ea030003-d4e5-4f60-8a03-000000000003\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T13:07:57.060Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"eb030003-e5f6-4071-9b03-000000000003\",\r\n \"Id\": 715010303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T13:07:55.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T13:07:56.110Z\",\r\n \"Message\": \"[Read CSV] Read 30 rows from data\\\\staging.csv\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T13:07:56.910Z\",\r\n \"Message\": \"[Write CSV] Access to the path 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvPublish\\\\data\\\\published.csv' is denied.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T13:07:56.980Z\",\r\n \"Message\": \"Write CSV: Access to the path 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\CsvPublish\\\\data\\\\published.csv' is denied.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T13:07:57.030Z\",\r\n \"Message\": \"Execution ended in: 00:00:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ea030003-d4e5-4f60-8a03-000000000003\",\r\n \"Name\": \"Outputs\",\r\n \"Path\": \"Outputs\",\r\n \"Description\": \"Production folder for CSV publishing processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/task.yaml index 47f087bbd5..bd140e1275 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-access-denied/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/README.md new file mode 100644 index 0000000000..457ec0be4e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/README.md @@ -0,0 +1,44 @@ +# Write CSV Failure - "Cannot set unknown member" (Package Version Skew) + +This scenario reproduces a `Write CSV` failure caused by **activity-package +version skew** between the build machine and the runtime robot. The project +targets `UiPath.System.Activities 25.10.5` and sets the Write CSV `Encoding` +property, but the robot runs an **older** version whose `WriteCsvFile` has no +`Encoding` member — so the workflow fails to initialize with `Cannot set unknown +member 'UiPath.Core.Activities.WriteCsvFile.Encoding'`. + +## What this scenario uncovers + +**Root Cause:** The runtime robot's `UiPath.System.Activities` is older than the +version the workflow was built with, so a property present in the build-time +activity is "unknown" to the runtime activity during XAML deserialization. + +This maps to: +`references/activity-packages/csv-activities/playbooks/write-csv-cannot-set-unknown-member.md` + +"Works in Studio, fails on the robot" is the signature. The fix is to align the +runtime's package versions with `project.json` (or rebuild against the runtime's +versions) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project pinning `UiPath.System.Activities 25.10.5`, with a `Write CSV` that sets `Encoding=UTF-8` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the build-vs-runtime + activity-package version skew (a property unknown to the robot's older + activity) and recommends aligning the robot's package versions with + `project.json` (or rebuilding against the runtime versions), without + fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/README.md deleted file mode 100644 index 8fb90ec802..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Write CSV Failure - "Cannot set unknown member" (Package Version Skew) - -This scenario reproduces a `Write CSV` failure caused by **activity-package -version skew** between the build machine and the runtime robot. The project -targets `UiPath.System.Activities 25.10.5` and sets the Write CSV `Encoding` -property, but the robot runs an **older** version whose `WriteCsvFile` has no -`Encoding` member — so the workflow fails to initialize with `Cannot set unknown -member 'UiPath.Core.Activities.WriteCsvFile.Encoding'`. - -## What this scenario uncovers - -**Root Cause:** The runtime robot's `UiPath.System.Activities` is older than the -version the workflow was built with, so a property present in the build-time -activity is "unknown" to the runtime activity during XAML deserialization. - -This maps to: -`references/activity-packages/csv-activities/playbooks/write-csv-cannot-set-unknown-member.md` - -"Works in Studio, fails on the robot" is the signature. The fix is to align the -runtime's package versions with `project.json` (or rebuild against the runtime's -versions) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project pinning `UiPath.System.Activities 25.10.5`, with a `Write CSV` that sets `Encoding=UTF-8` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the job faults at workflow init with the deserialization error | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the build-vs-runtime - activity-package version skew (a property unknown to the robot's older - activity) and recommends aligning the robot's package versions with - `project.json` (or rebuilding against the runtime versions), without - fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/manifest.json deleted file mode 100644 index 62c0430560..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ea010001-d4e5-4f60-8a01-000000000001\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea010001-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ea010001-d4e5-4f60-8a01-000000000001 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea010001-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"eb010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-get-eb010001-output-json.json" - }, - { - "match": "or jobs get eb010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-get-eb010001-output-json.json" - }, - { - "match": "or jobs logs \"eb010001-e5f6-4071-9b01-000000000001\" --level Error --output json", - "file": "or-jobs-logs-eb010001-level-error-output-json.json" - }, - { - "match": "or jobs logs eb010001-e5f6-4071-9b01-000000000001 --level Error --output json", - "file": "or-jobs-logs-eb010001-level-error-output-json.json" - }, - { - "match": "or jobs logs \"eb010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-logs-eb010001-output-json.json" - }, - { - "match": "or jobs logs eb010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-logs-eb010001-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 159097a99f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ea010001-d4e5-4f60-8a01-000000000001", - "Name": "Exports", - "Path": "Exports", - "Description": "Production folder for report-export processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-get-eb010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-get-eb010001-output-json.json deleted file mode 100644 index 5ca57d7b3b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-get-eb010001-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "eb010001-e5f6-4071-9b01-000000000001", - "StartTime": "2026-06-15T08:20:11.140Z", - "EndTime": "2026-06-15T08:20:12.760Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ec010001-f6a7-4182-9c01-000000000001", - "Info": "Error initializing workflow 'Main.xaml': Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\r\n\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Main \"Main\"\r\n\r\nSystem.Xaml.XamlObjectWriterException: Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.", - "CreationTime": "2026-06-15T08:20:10.700Z", - "ReleaseName": "ReportExport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7015001, - "OrganizationUnitId": 7791618, - "FolderKey": "ea010001-d4e5-4f60-8a01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T08:20:12.760Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "eb010001-e5f6-4071-9b01-000000000001", - "Id": 715010101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-list-folder-key-ea010001-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-list-folder-key-ea010001-state-faulted-output-json.json deleted file mode 100644 index b57d3eaccf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-list-folder-key-ea010001-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "eb010001-e5f6-4071-9b01-000000000001", - "State": "Faulted", - "ProcessName": "ReportExport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T08:20:11.140Z", - "EndTime": "2026-06-15T08:20:12.760Z", - "CreationTime": "2026-06-15T08:20:10.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-logs-eb010001-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-logs-eb010001-level-error-output-json.json deleted file mode 100644 index 2765cd1433..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-logs-eb010001-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T08:20:12.610Z", - "Message": "Error initializing workflow 'Main.xaml': Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T08:20:12.680Z", - "Message": "Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-logs-eb010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-logs-eb010001-output-json.json deleted file mode 100644 index c5764c8357..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/m/r/or-jobs-logs-eb010001-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T08:20:11.140Z", - "Message": "Execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T08:20:12.610Z", - "Message": "Error initializing workflow 'Main.xaml': Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T08:20:12.680Z", - "Message": "Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T08:20:12.740Z", - "Message": "Execution ended in: 00:00:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/uip-fixture.json new file mode 100644 index 0000000000..cd6765d486 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ea010001-d4e5-4f60-8a01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"eb010001-e5f6-4071-9b01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T08:20:11.140Z\",\r\n \"EndTime\": \"2026-06-15T08:20:12.760Z\",\r\n \"CreationTime\": \"2026-06-15T08:20:10.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb010001-e5f6-4071-9b01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T08:20:12.610Z\",\r\n \"Message\": \"Error initializing workflow 'Main.xaml': Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T08:20:12.680Z\",\r\n \"Message\": \"Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "eb010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"eb010001-e5f6-4071-9b01-000000000001\",\r\n \"StartTime\": \"2026-06-15T08:20:11.140Z\",\r\n \"EndTime\": \"2026-06-15T08:20:12.760Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ec010001-f6a7-4182-9c01-000000000001\",\r\n \"Info\": \"Error initializing workflow 'Main.xaml': Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\\r\\n\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Xaml.XamlObjectWriterException: Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\",\r\n \"CreationTime\": \"2026-06-15T08:20:10.700Z\",\r\n \"ReleaseName\": \"ReportExport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7015001,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ea010001-d4e5-4f60-8a01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T08:20:12.760Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"eb010001-e5f6-4071-9b01-000000000001\",\r\n \"Id\": 715010101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T08:20:11.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T08:20:12.610Z\",\r\n \"Message\": \"Error initializing workflow 'Main.xaml': Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T08:20:12.680Z\",\r\n \"Message\": \"Cannot set unknown member 'UiPath.Core.Activities.WriteCsvFile.Encoding'.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T08:20:12.740Z\",\r\n \"Message\": \"Execution ended in: 00:00:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ea010001-d4e5-4f60-8a01-000000000001\",\r\n \"Name\": \"Exports\",\r\n \"Path\": \"Exports\",\r\n \"Description\": \"Production folder for report-export processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/task.yaml index 37ea847526..a296a5a2df 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-cannot-set-unknown-member/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/README.md new file mode 100644 index 0000000000..6338ef9fa5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/README.md @@ -0,0 +1,40 @@ +# Write CSV Failure - "Failed to create a 'Delimitator' from the text 'Tab'" + +This scenario reproduces a `Write CSV` failure where the `Delimiter` property is +bound to a string variable holding the **word** `"Tab"` instead of the enum value +or a character literal. The activity can't map the word to a real delimiter and +faults with `Failed to create a 'Delimitator' from the text 'Tab'`. + +## What this scenario uncovers + +**Root Cause:** `Delimiter="[delimiterName]"` where `delimiterName` is a `String` += `"Tab"`. The activity expects the delimiter enum / a character, not the +localized word. + +This maps to: +`references/activity-packages/csv-activities/playbooks/write-csv-invalid-delimiter.md` + +The fix is a property change (pick from the dropdown, or pass the character +literal `vbTab`/`"\t"`) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Write CSV` whose `Delimiter` is bound to a `String` variable `delimiterName = "Tab"` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the Delimiter set from a + string/word (`"Tab"`) rather than the enum/character literal, and recommends + selecting the delimiter from the dropdown or passing the character literal + (`vbTab` / `"\t"`), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/README.md deleted file mode 100644 index ad07a359f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Write CSV Failure - "Failed to create a 'Delimitator' from the text 'Tab'" - -This scenario reproduces a `Write CSV` failure where the `Delimiter` property is -bound to a string variable holding the **word** `"Tab"` instead of the enum value -or a character literal. The activity can't map the word to a real delimiter and -faults with `Failed to create a 'Delimitator' from the text 'Tab'`. - -## What this scenario uncovers - -**Root Cause:** `Delimiter="[delimiterName]"` where `delimiterName` is a `String` -= `"Tab"`. The activity expects the delimiter enum / a character, not the -localized word. - -This maps to: -`references/activity-packages/csv-activities/playbooks/write-csv-invalid-delimiter.md` - -The fix is a property change (pick from the dropdown, or pass the character -literal `vbTab`/`"\t"`) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Write CSV` whose `Delimiter` is bound to a `String` variable `delimiterName = "Tab"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the Delimiter set from a - string/word (`"Tab"`) rather than the enum/character literal, and recommends - selecting the delimiter from the dropdown or passing the character literal - (`vbTab` / `"\t"`), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/manifest.json deleted file mode 100644 index 233076bf35..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ea020002-d4e5-4f60-8a02-000000000002\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea020002-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ea020002-d4e5-4f60-8a02-000000000002 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea020002-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"eb020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-get-eb020002-output-json.json" - }, - { - "match": "or jobs get eb020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-get-eb020002-output-json.json" - }, - { - "match": "or jobs logs \"eb020002-e5f6-4071-9b02-000000000002\" --level Error --output json", - "file": "or-jobs-logs-eb020002-level-error-output-json.json" - }, - { - "match": "or jobs logs eb020002-e5f6-4071-9b02-000000000002 --level Error --output json", - "file": "or-jobs-logs-eb020002-level-error-output-json.json" - }, - { - "match": "or jobs logs \"eb020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-logs-eb020002-output-json.json" - }, - { - "match": "or jobs logs eb020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-logs-eb020002-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 233a50674e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ea020002-d4e5-4f60-8a02-000000000002", - "Name": "Extracts", - "Path": "Extracts", - "Description": "Production folder for delimited-export processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-get-eb020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-get-eb020002-output-json.json deleted file mode 100644 index 160b673e1b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-get-eb020002-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "eb020002-e5f6-4071-9b02-000000000002", - "StartTime": "2026-06-15T10:33:24.140Z", - "EndTime": "2026-06-15T10:33:26.020Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ec020002-f6a7-4182-9c02-000000000002", - "Info": "Write CSV: Failed to create a 'Delimitator' from the text 'Tab'.\r\n\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ComponentModel.FormatException: Failed to create a 'Delimitator' from the text 'Tab'.", - "CreationTime": "2026-06-15T10:33:23.700Z", - "ReleaseName": "DelimitedExport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7015002, - "OrganizationUnitId": 7791618, - "FolderKey": "ea020002-d4e5-4f60-8a02-000000000002", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T10:33:26.020Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "eb020002-e5f6-4071-9b02-000000000002", - "Id": 715010202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-list-folder-key-ea020002-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-list-folder-key-ea020002-state-faulted-output-json.json deleted file mode 100644 index b7e9867a4a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-list-folder-key-ea020002-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "eb020002-e5f6-4071-9b02-000000000002", - "State": "Faulted", - "ProcessName": "DelimitedExport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T10:33:24.140Z", - "EndTime": "2026-06-15T10:33:26.020Z", - "CreationTime": "2026-06-15T10:33:23.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-logs-eb020002-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-logs-eb020002-level-error-output-json.json deleted file mode 100644 index 4187e9f6ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-logs-eb020002-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T10:33:25.880Z", - "Message": "[Write CSV] Failed to create a 'Delimitator' from the text 'Tab'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T10:33:25.950Z", - "Message": "Write CSV: Failed to create a 'Delimitator' from the text 'Tab'.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-logs-eb020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-logs-eb020002-output-json.json deleted file mode 100644 index 9ac08587af..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/m/r/or-jobs-logs-eb020002-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T10:33:24.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T10:33:25.110Z", - "Message": "[Read CSV] Read 18 rows from data\\source.csv" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T10:33:25.880Z", - "Message": "[Write CSV] Failed to create a 'Delimitator' from the text 'Tab'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T10:33:25.950Z", - "Message": "Write CSV: Failed to create a 'Delimitator' from the text 'Tab'.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T10:33:25.990Z", - "Message": "Execution ended in: 00:00:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/uip-fixture.json new file mode 100644 index 0000000000..648941c994 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ea020002-d4e5-4f60-8a02-000000000002", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"eb020002-e5f6-4071-9b02-000000000002\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DelimitedExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T10:33:24.140Z\",\r\n \"EndTime\": \"2026-06-15T10:33:26.020Z\",\r\n \"CreationTime\": \"2026-06-15T10:33:23.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb020002-e5f6-4071-9b02-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T10:33:25.880Z\",\r\n \"Message\": \"[Write CSV] Failed to create a 'Delimitator' from the text 'Tab'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T10:33:25.950Z\",\r\n \"Message\": \"Write CSV: Failed to create a 'Delimitator' from the text 'Tab'.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "eb020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"eb020002-e5f6-4071-9b02-000000000002\",\r\n \"StartTime\": \"2026-06-15T10:33:24.140Z\",\r\n \"EndTime\": \"2026-06-15T10:33:26.020Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ec020002-f6a7-4182-9c02-000000000002\",\r\n \"Info\": \"Write CSV: Failed to create a 'Delimitator' from the text 'Tab'.\\r\\n\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ComponentModel.FormatException: Failed to create a 'Delimitator' from the text 'Tab'.\",\r\n \"CreationTime\": \"2026-06-15T10:33:23.700Z\",\r\n \"ReleaseName\": \"DelimitedExport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7015002,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ea020002-d4e5-4f60-8a02-000000000002\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T10:33:26.020Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"eb020002-e5f6-4071-9b02-000000000002\",\r\n \"Id\": 715010202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T10:33:24.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T10:33:25.110Z\",\r\n \"Message\": \"[Read CSV] Read 18 rows from data\\\\source.csv\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T10:33:25.880Z\",\r\n \"Message\": \"[Write CSV] Failed to create a 'Delimitator' from the text 'Tab'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T10:33:25.950Z\",\r\n \"Message\": \"Write CSV: Failed to create a 'Delimitator' from the text 'Tab'.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T10:33:25.990Z\",\r\n \"Message\": \"Execution ended in: 00:00:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ea020002-d4e5-4f60-8a02-000000000002\",\r\n \"Name\": \"Extracts\",\r\n \"Path\": \"Extracts\",\r\n \"Description\": \"Production folder for delimited-export processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/task.yaml index 55f9939ac1..b0aabe2ca1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-invalid-delimiter/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/README.md new file mode 100644 index 0000000000..b627a98c4b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/README.md @@ -0,0 +1,38 @@ +# Write CSV Failure - "Unsupported encoding name" + +This scenario reproduces a `Write CSV` failure where the `Encoding` property is +set to a string .NET does not recognize (`utf8-bom`), so the activity faults with +`'utf8-bom' is not a supported encoding name` (`ArgumentException`). + +## What this scenario uncovers + +**Root Cause:** `Encoding="utf8-bom"` is not a valid .NET encoding name. The +activity calls `Encoding.GetEncoding("utf8-bom")`, which throws. + +This maps to: +`references/activity-packages/csv-activities/playbooks/write-csv-unsupported-encoding.md` + +The fix is a property change (use a valid .NET encoding name, leave it default, +or use Output Data Table → Write Text File) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Write CSV` whose `Encoding=utf8-bom` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the invalid `Encoding` + string and recommends using a valid .NET encoding name (`UTF-8` / + `Windows-1252`), leaving it default, or Output Data Table → Write Text File, + without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/README.md deleted file mode 100644 index a967d169c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# Write CSV Failure - "Unsupported encoding name" - -This scenario reproduces a `Write CSV` failure where the `Encoding` property is -set to a string .NET does not recognize (`utf8-bom`), so the activity faults with -`'utf8-bom' is not a supported encoding name` (`ArgumentException`). - -## What this scenario uncovers - -**Root Cause:** `Encoding="utf8-bom"` is not a valid .NET encoding name. The -activity calls `Encoding.GetEncoding("utf8-bom")`, which throws. - -This maps to: -`references/activity-packages/csv-activities/playbooks/write-csv-unsupported-encoding.md` - -The fix is a property change (use a valid .NET encoding name, leave it default, -or use Output Data Table → Write Text File) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Write CSV` whose `Encoding=utf8-bom` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the invalid `Encoding` - string and recommends using a valid .NET encoding name (`UTF-8` / - `Windows-1252`), leaving it default, or Output Data Table → Write Text File, - without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/manifest.json deleted file mode 100644 index c5693d3db1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"ea040004-d4e5-4f60-8a04-000000000004\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea040004-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key ea040004-d4e5-4f60-8a04-000000000004 --state Faulted --output json", - "file": "or-jobs-list-folder-key-ea040004-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"eb040004-e5f6-4071-9b04-000000000004\" --output json", - "file": "or-jobs-get-eb040004-output-json.json" - }, - { - "match": "or jobs get eb040004-e5f6-4071-9b04-000000000004 --output json", - "file": "or-jobs-get-eb040004-output-json.json" - }, - { - "match": "or jobs logs \"eb040004-e5f6-4071-9b04-000000000004\" --level Error --output json", - "file": "or-jobs-logs-eb040004-level-error-output-json.json" - }, - { - "match": "or jobs logs eb040004-e5f6-4071-9b04-000000000004 --level Error --output json", - "file": "or-jobs-logs-eb040004-level-error-output-json.json" - }, - { - "match": "or jobs logs \"eb040004-e5f6-4071-9b04-000000000004\" --output json", - "file": "or-jobs-logs-eb040004-output-json.json" - }, - { - "match": "or jobs logs eb040004-e5f6-4071-9b04-000000000004 --output json", - "file": "or-jobs-logs-eb040004-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 3e58845b81..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "ea040004-d4e5-4f60-8a04-000000000004", - "Name": "Encoders", - "Path": "Encoders", - "Description": "Production folder for CSV encoding processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-get-eb040004-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-get-eb040004-output-json.json deleted file mode 100644 index 7d84c4b32a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-get-eb040004-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "eb040004-e5f6-4071-9b04-000000000004", - "StartTime": "2026-06-15T15:18:42.140Z", - "EndTime": "2026-06-15T15:18:44.010Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ec040004-f6a7-4182-9c04-000000000004", - "Info": "Write CSV: 'utf8-bom' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method. (Parameter 'name')\r\n\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: 'utf8-bom' is not a supported encoding name.", - "CreationTime": "2026-06-15T15:18:41.700Z", - "ReleaseName": "CsvEncode", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7015004, - "OrganizationUnitId": 7791618, - "FolderKey": "ea040004-d4e5-4f60-8a04-000000000004", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T15:18:44.010Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "eb040004-e5f6-4071-9b04-000000000004", - "Id": 715010404 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-list-folder-key-ea040004-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-list-folder-key-ea040004-state-faulted-output-json.json deleted file mode 100644 index 25dce50adf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-list-folder-key-ea040004-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "eb040004-e5f6-4071-9b04-000000000004", - "State": "Faulted", - "ProcessName": "CsvEncode", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T15:18:42.140Z", - "EndTime": "2026-06-15T15:18:44.010Z", - "CreationTime": "2026-06-15T15:18:41.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-logs-eb040004-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-logs-eb040004-level-error-output-json.json deleted file mode 100644 index 417dedbc1f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-logs-eb040004-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T15:18:43.870Z", - "Message": "[Write CSV] 'utf8-bom' is not a supported encoding name. (Parameter 'name')" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T15:18:43.940Z", - "Message": "Write CSV: 'utf8-bom' is not a supported encoding name.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-logs-eb040004-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-logs-eb040004-output-json.json deleted file mode 100644 index 9c18ea57a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/m/r/or-jobs-logs-eb040004-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T15:18:42.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T15:18:43.090Z", - "Message": "[Read CSV] Read 25 rows from data\\source.csv" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T15:18:43.870Z", - "Message": "[Write CSV] 'utf8-bom' is not a supported encoding name. (Parameter 'name')" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T15:18:43.940Z", - "Message": "Write CSV: 'utf8-bom' is not a supported encoding name.\r\nin Main.xaml\r\n at WriteCsvFile \"Write CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T15:18:43.980Z", - "Message": "Execution ended in: 00:00:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/uip-fixture.json new file mode 100644 index 0000000000..6fac20ba80 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "ea040004-d4e5-4f60-8a04-000000000004", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"eb040004-e5f6-4071-9b04-000000000004\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CsvEncode\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T15:18:42.140Z\",\r\n \"EndTime\": \"2026-06-15T15:18:44.010Z\",\r\n \"CreationTime\": \"2026-06-15T15:18:41.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb040004-e5f6-4071-9b04-000000000004", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T15:18:43.870Z\",\r\n \"Message\": \"[Write CSV] 'utf8-bom' is not a supported encoding name. (Parameter 'name')\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T15:18:43.940Z\",\r\n \"Message\": \"Write CSV: 'utf8-bom' is not a supported encoding name.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "eb040004-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"eb040004-e5f6-4071-9b04-000000000004\",\r\n \"StartTime\": \"2026-06-15T15:18:42.140Z\",\r\n \"EndTime\": \"2026-06-15T15:18:44.010Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ec040004-f6a7-4182-9c04-000000000004\",\r\n \"Info\": \"Write CSV: 'utf8-bom' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method. (Parameter 'name')\\r\\n\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: 'utf8-bom' is not a supported encoding name.\",\r\n \"CreationTime\": \"2026-06-15T15:18:41.700Z\",\r\n \"ReleaseName\": \"CsvEncode\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7015004,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"ea040004-d4e5-4f60-8a04-000000000004\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T15:18:44.010Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"eb040004-e5f6-4071-9b04-000000000004\",\r\n \"Id\": 715010404\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "eb040004-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T15:18:42.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T15:18:43.090Z\",\r\n \"Message\": \"[Read CSV] Read 25 rows from data\\\\source.csv\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T15:18:43.870Z\",\r\n \"Message\": \"[Write CSV] 'utf8-bom' is not a supported encoding name. (Parameter 'name')\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T15:18:43.940Z\",\r\n \"Message\": \"Write CSV: 'utf8-bom' is not a supported encoding name.\\r\\nin Main.xaml\\r\\n at WriteCsvFile \\\"Write CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T15:18:43.980Z\",\r\n \"Message\": \"Execution ended in: 00:00:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ea040004-d4e5-4f60-8a04-000000000004\",\r\n \"Name\": \"Encoders\",\r\n \"Path\": \"Encoders\",\r\n \"Description\": \"Production folder for CSV encoding processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/task.yaml index e6b55abc72..60256c1ed3 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/csv-write-unsupported-encoding/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/README.md new file mode 100644 index 0000000000..e510c2609f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/README.md @@ -0,0 +1,74 @@ +# CV Click — ElementNotFound (descriptor no longer matches the screen) — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim `uip` CLI +responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, process **CV** (key `37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9`), faulted with +`UiPath.CV.ElementNotFoundException: Element not found` thrown from the +Computer Vision **CV Click** activity (`CvClickWithDescriptor`, "CV Click - +'Button'") inside the **CV Screen Scope** (`CVScope`, "CV Screen Scope +'msedge.exe Google'") in `InvoiceEntryUI.xaml`. The find could not match +its locked descriptor geometry against the live `msedge.exe / Google` screen +and the retry loop exhausted its timeout — timeout expiry surfaces AS +`ElementNotFoundException` (there is no separate `TimeoutException`). + +### Verdict + +**Branch A — genuine descriptor mismatch — is the operative cause:** the +captured descriptor (target Button + two anchor Buttons) no longer matches the +current page, so CV returns no matching region within `TimeoutMS`. **Branch B** +(scope root lost / screenshot-refresh failure) cannot be fully excluded from +job-level evidence alone — confirming it requires the host-side +`*_ComputerVision` runtime dump on the run machine (unavailable from this +session). The fix is to re-indicate the CV Click target in Studio to regenerate +the descriptor — **not** to merely raise `TimeoutMS` (geometry/match problem, +not time) — after confirming Branch A vs B via the dump. + +Ruled out: invalid descriptor, scroll-search exhaustion, cell-targeting, CV +server / auth / throttling / network, scope-setup failure, action-failed-after-find. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of the failing `CV` project — `project.json` + `InvoiceEntryUI.xaml` only | + +### Trace spans + +One `RobotJob` span was recorded for this job; both trace-retrieval forms +(`or jobs traces ` and `traces spans get --job-key `) replay it. Its +`Attributes` carry the full `UiPath.CV.ElementNotFoundException` .NET stack with +the `CvClickWithDescriptor.EndExecute` faulting frame propagating through +`CVScope.OnFault` — confirming the fault is the CV Click find, not scope setup +or a downstream action. No per-activity child spans were emitted, so the trace +adds the verbatim stack but no attribute-level disambiguation beyond `jobs get`. + +### Fixture derivations and scrubs + +- `or-jobs-logs-...-level-error.json` is the recorded Error-level log query (the + only logs raw captured); the generic logs rule maps to the same fixture — the + exception row is the only diagnostic content. +- No `--state Faulted` jobs-list raw was recorded, so the folder-key rule + substring-matches `--state Faulted` variants and returns the full recorded + list. The failing CV job is the most recent row. +- Standard scrubs applied: run-machine hostname → `MOCK-HOST`; workspace / + Orchestrator-identity email → `original_email@test.com`; local Windows account + → `UIPATH\REPLACEMENT_USER`; the CV Screen Scope `ApiKey` literal in the + snapshotted `InvoiceEntryUI.xaml` → `MOCK_CV_API_KEY`. Job and folder keys + and the `Element not found` / `ElementNotFoundException` error text are kept + verbatim. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/1297a9f862.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/1297a9f862.json deleted file mode 100644 index 35adb9286c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/1297a9f862.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-12T07:27:31.223Z", - "JobId": 66807528 - }, - { - "State": "Running", - "Timestamp": "2026-06-12T07:26:50.023Z", - "JobId": 66807528 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/330a08b85f.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/330a08b85f.json deleted file mode 100644 index abb8a79ca9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/330a08b85f.json +++ /dev/null @@ -1,572 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "State": "Faulted", - "ProcessName": "CV", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T07:26:49.563Z", - "EndTime": "2026-06-12T07:27:30.893Z", - "CreationTime": "2026-06-12T07:26:49.563Z" - }, - { - "Key": "7cb89ff3-dd13-452d-b194-bba867b0e4d9", - "State": "Stopped", - "ProcessName": "CV", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-06-12T07:26:32.810Z", - "EndTime": "2026-06-12T07:26:32.810Z", - "CreationTime": "2026-06-12T07:26:17.090Z" - }, - { - "Key": "e4237b15-1574-4c7a-8657-fe0b0be2aa75", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-05T07:34:54.547Z", - "EndTime": "2026-06-05T07:34:56.163Z", - "CreationTime": "2026-06-05T07:34:54.547Z" - }, - { - "Key": "f32f9829-4465-49a2-8e3e-0cc182a020c4", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T12:57:51.543Z", - "EndTime": "2026-06-04T12:57:52.790Z", - "CreationTime": "2026-06-04T12:57:51.543Z" - }, - { - "Key": "779be59f-44a2-4447-8105-3306a168813f", - "State": "Stopped", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T12:55:26.010Z", - "EndTime": "2026-06-04T12:55:34.487Z", - "CreationTime": "2026-06-04T12:55:26.010Z" - }, - { - "Key": "35c669cb-7f46-4282-b90d-ca7abd52c59f", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:50:06.750Z", - "EndTime": "2026-06-04T10:50:12.563Z", - "CreationTime": "2026-06-04T10:50:06.750Z" - }, - { - "Key": "b1cab28d-5ab7-42f1-98f3-1920b637ea06", - "State": "Faulted", - "ProcessName": "Legacy", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T10:42:10.817Z", - "EndTime": "2026-06-04T10:42:43.897Z", - "CreationTime": "2026-06-04T10:42:10.817Z" - }, - { - "Key": "8e2c5979-e05f-4e03-b6d1-cda02aa8450c", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T15:10:44.610Z", - "EndTime": "2026-06-03T15:11:17.967Z", - "CreationTime": "2026-06-03T15:10:44.610Z" - }, - { - "Key": "cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T11:56:44.780Z", - "EndTime": "2026-06-03T11:56:52.850Z", - "CreationTime": "2026-06-03T11:56:44.780Z" - }, - { - "Key": "025d0847-ce15-494f-91d6-04be2a19be6d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:28:08.133Z", - "EndTime": "2026-06-03T09:29:10.690Z", - "CreationTime": "2026-06-03T09:28:08.133Z" - }, - { - "Key": "90058dec-f287-4b4d-8272-1b7cf72e0c0d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:17:53.650Z", - "EndTime": "2026-06-03T09:18:57.633Z", - "CreationTime": "2026-06-03T09:17:53.650Z" - }, - { - "Key": "f215a846-a117-4c56-be2a-28a2d6894cac", - "State": "Successful", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T09:41:25.187Z", - "EndTime": "2026-06-02T09:41:53.797Z", - "CreationTime": "2026-06-02T09:41:25.187Z" - }, - { - "Key": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T07:30:38.093Z", - "EndTime": "2026-06-02T07:31:37.367Z", - "CreationTime": "2026-06-02T07:30:38.093Z" - }, - { - "Key": "68d80da7-719f-44cd-b7d6-94be679f55d4", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:52:06.580Z", - "EndTime": "2026-06-02T06:52:40.293Z", - "CreationTime": "2026-06-02T06:52:06.580Z" - }, - { - "Key": "3975787c-87d9-4927-bc7c-a19165e06035", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:42:21.737Z", - "EndTime": "2026-06-02T06:42:56.063Z", - "CreationTime": "2026-06-02T06:42:21.737Z" - }, - { - "Key": "b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d", - "State": "Pending", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-29T12:28:56.660Z" - }, - { - "Key": "ccc3e841-bdcf-4501-9d30-f112f098f630", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T13:09:18.353Z", - "EndTime": "2026-05-28T13:10:10.310Z", - "CreationTime": "2026-05-28T12:48:07.547Z" - }, - { - "Key": "cb00355c-3835-4c18-b37e-af5c7609c599", - "State": "Successful", - "ProcessName": "NewBlankTask", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T10:28:07.327Z", - "EndTime": "2026-05-28T10:29:16.540Z", - "CreationTime": "2026-05-28T08:48:18.580Z" - }, - { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T08:00:50.870Z", - "EndTime": "2026-05-28T08:01:02.417Z", - "CreationTime": "2026-05-27T16:24:24.603Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-26T08:48:20.617Z", - "EndTime": "2026-05-26T08:48:20.617Z", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - } - ], - "Pagination": { - "Returned": 40, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/93706b31af.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/93706b31af.json deleted file mode 100644 index 41aab1b115..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/93706b31af.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-f493-d5c20a3c0ca5", - "TraceId": "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "ParentId": null, - "Name": "Robot Job 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "StartTime": "2026-06-12T07:26:49.8092821Z", - "EndTime": "2026-06-12T07:27:30.482966Z", - "Attributes": "{\"Exception\":\"UiPath.CV.ElementNotFoundException: Element not found\\r\\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "c81daadb-3ad3-4681-9fd8-627e57742f4b", - "TenantId": "3ea4784c-f5f9-4c9f-9cd9-a81c64569726", - "ExpiryTimeUtc": null, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "b4a18188-e61d-4786-9f5f-3dc4b8f2c111", - "JobKey": "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-12T07:27:30.7782589Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/9b8544165d.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/9b8544165d.json deleted file mode 100644 index b43282968f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/9b8544165d.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "StartTime": "2026-06-12T07:26:49.563Z", - "EndTime": "2026-06-12T07:27:30.893Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "Info": "Element not found\r\n\r\nin InvoiceEntryUI.xaml\r\n at CvClickWithDescriptor \"CV Click - 'Button'\"\r\n at Sequence \"Do\"\r\n at CVScope \"CV Screen Scope 'msedge.exe Google'\"\r\n at Sequence \"CV Element Not Found repro\"\r\n at InvoiceEntryUI \"InvoiceEntryUI\"\r\n\r\nUiPath.CV.ElementNotFoundException: Element not found\r\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-12T07:26:49.563Z", - "ReleaseName": "CV", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21107282, - "EntryPointPath": "InvoiceEntryUI.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T07:27:30.893Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "37e7a8bbd2074db48ebff0ccc7aa85e9", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 66807528 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/README.md deleted file mode 100644 index 6a2a0c1c5e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/README.md +++ /dev/null @@ -1,73 +0,0 @@ -# CV Click — ElementNotFound (descriptor no longer matches the screen) — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim `uip` CLI -responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, process **CV** (key `37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9`), faulted with -`UiPath.CV.ElementNotFoundException: Element not found` thrown from the -Computer Vision **CV Click** activity (`CvClickWithDescriptor`, "CV Click - -'Button'") inside the **CV Screen Scope** (`CVScope`, "CV Screen Scope -'msedge.exe Google'") in `InvoiceEntryUI.xaml`. The find could not match -its locked descriptor geometry against the live `msedge.exe / Google` screen -and the retry loop exhausted its timeout — timeout expiry surfaces AS -`ElementNotFoundException` (there is no separate `TimeoutException`). - -### Verdict - -**Branch A — genuine descriptor mismatch — is the operative cause:** the -captured descriptor (target Button + two anchor Buttons) no longer matches the -current page, so CV returns no matching region within `TimeoutMS`. **Branch B** -(scope root lost / screenshot-refresh failure) cannot be fully excluded from -job-level evidence alone — confirming it requires the host-side -`*_ComputerVision` runtime dump on the run machine (unavailable from this -session). The fix is to re-indicate the CV Click target in Studio to regenerate -the descriptor — **not** to merely raise `TimeoutMS` (geometry/match problem, -not time) — after confirming Branch A vs B via the dump. - -Ruled out: invalid descriptor, scroll-search exhaustion, cell-targeting, CV -server / auth / throttling / network, scope-setup failure, action-failed-after-find. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | -| `process/` | snapshot of the failing `CV` project — `project.json` + `InvoiceEntryUI.xaml` only | - -### Trace spans - -One `RobotJob` span was recorded for this job; both trace-retrieval forms -(`or jobs traces ` and `traces spans get --job-key `) replay it. Its -`Attributes` carry the full `UiPath.CV.ElementNotFoundException` .NET stack with -the `CvClickWithDescriptor.EndExecute` faulting frame propagating through -`CVScope.OnFault` — confirming the fault is the CV Click find, not scope setup -or a downstream action. No per-activity child spans were emitted, so the trace -adds the verbatim stack but no attribute-level disambiguation beyond `jobs get`. - -### Fixture derivations and scrubs - -- `or-jobs-logs-...-level-error.json` is the recorded Error-level log query (the - only logs raw captured); the generic logs rule maps to the same fixture — the - exception row is the only diagnostic content. -- No `--state Faulted` jobs-list raw was recorded, so the folder-key rule - substring-matches `--state Faulted` variants and returns the full recorded - list. The failing CV job is the most recent row. -- Standard scrubs applied: run-machine hostname → `MOCK-HOST`; workspace / - Orchestrator-identity email → `original_email@test.com`; local Windows account - → `UIPATH\REPLACEMENT_USER`; the CV Screen Scope `ApiKey` literal in the - snapshotted `InvoiceEntryUI.xaml` → `MOCK_CV_API_KEY`. Job and folder keys - and the `Element not found` / `ElementNotFoundException` error text are kept - verbatim. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/b538fa3653.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/b538fa3653.json deleted file mode 100644 index 53109aa1c1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/b538fa3653.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-12T07:27:30.476Z", - "Message": "CV Click - 'Button': Element not found\r\nin InvoiceEntryUI.xaml\r\n at CvClickWithDescriptor \"CV Click - 'Button'\"\r\n at Sequence \"Do\"\r\n at CVScope \"CV Screen Scope 'msedge.exe Google'\"\r\n at Sequence \"CV Element Not Found repro\"\r\n at InvoiceEntryUI \"InvoiceEntryUI\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.CV.ElementNotFoundException\r\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/manifest.json deleted file mode 100644 index 84ff7ee04b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,91 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "525de3e550.json" - }, - { - "match": "or jobs get 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "file": "9b8544165d.json" - }, - { - "match": "or jobs logs 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9 --level Error", - "file": "b538fa3653.json", - "_doc": "Recorded verbatim. The session captured only the Error-level log query; the single row is the job's one Error entry — the CV Click ElementNotFoundException." - }, - { - "match": "or jobs logs 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "file": "b538fa3653.json", - "_doc": "No unfiltered-logs raw was recorded; the Error-level set answers any logs variant — the exception row is the only diagnostic content." - }, - { - "match": "or jobs history 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "file": "1297a9f862.json", - "_doc": "Recorded verbatim: the Running -> Faulted transition set. Confirms a clean single execution — carries no diagnostic signal beyond jobs get." - }, - { - "match": "or jobs traces 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "file": "93706b31af.json", - "_doc": "Recorded verbatim. One RobotJob span whose Attributes carry the full UiPath.CV.ElementNotFoundException stack with the CvClickWithDescriptor.EndExecute faulting frame and the CVScope.OnFault propagation frame. No child CvClick/CVScope spans were emitted (no per-activity OpenTelemetry spans for this job), so the trace adds the verbatim .NET stack but no extra attribute-level disambiguation beyond jobs get/logs." - }, - { - "match": "traces spans get --job-key 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "file": "93706b31af.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Maps to the same recorded span payload so the investigation does not depend on which trace form the agent reaches for." - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "330a08b85f.json", - "_doc": "No --state Faulted raw was recorded in this session, so there is no Faulted-only fixture; substring dispatch routes `--state Faulted` variants here too, returning the full recorded list — the failing CV job (37e7a8bb...) is the most recent row." - }, - { - "match": "or jobs list", - "file": "330a08b85f.json", - "_doc": "Fallback for flag-order variations. The Shared folder's recorded job list is the correct answer for any jobs-list variant the agent tries; the most recent row is the faulted CV job." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key (defb8e05-e36b-4c36-bf11-0b4d08ce6cd1) before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job — the CV process job 37e7a8bb..." - }, - { - "pattern": "or jobs get 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "min": 1, - "description": "Agent must fetch the failing job to read the UiPath.CV.ElementNotFoundException 'Element not found' fault from CvClickWithDescriptor (\"CV Click - 'Button'\") inside CVScope (\"CV Screen Scope 'msedge.exe Google'\") in InvoiceEntryUI.xaml." - }, - { - "pattern": "traces", - "min": 1, - "description": "Trace retrieval surfaces the verbatim .NET stack confirming the faulting frame is CvClickWithDescriptor.EndExecute propagating through CVScope.OnFault — pinning the fault to the CV Click find, not scope setup or a downstream action." - }, - { - "pattern": "or jobs logs 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", - "min": 0, - "description": "OPTIONAL — state transitions (Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..e410cd0427 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/data/uip-fixture.json @@ -0,0 +1,142 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T07:27:30.476Z\",\r\n \"Message\": \"CV Click - 'Button': Element not found\\r\\nin InvoiceEntryUI.xaml\\r\\n at CvClickWithDescriptor \\\"CV Click - 'Button'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at CVScope \\\"CV Screen Scope 'msedge.exe Google'\\\"\\r\\n at Sequence \\\"CV Element Not Found repro\\\"\\r\\n at InvoiceEntryUI \\\"InvoiceEntryUI\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.CV.ElementNotFoundException\\r\\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-f493-d5c20a3c0ca5\",\r\n \"TraceId\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"StartTime\": \"2026-06-12T07:26:49.8092821Z\",\r\n \"EndTime\": \"2026-06-12T07:27:30.482966Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.CV.ElementNotFoundException: Element not found\\\\r\\\\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\\\r\\\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"c81daadb-3ad3-4681-9fd8-627e57742f4b\",\r\n \"TenantId\": \"3ea4784c-f5f9-4c9f-9cd9-a81c64569726\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"b4a18188-e61d-4786-9f5f-3dc4b8f2c111\",\r\n \"JobKey\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-12T07:27:30.7782589Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CV\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T07:26:49.563Z\",\r\n \"EndTime\": \"2026-06-12T07:27:30.893Z\",\r\n \"CreationTime\": \"2026-06-12T07:26:49.563Z\"\r\n },\r\n {\r\n \"Key\": \"7cb89ff3-dd13-452d-b194-bba867b0e4d9\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"CV\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T07:26:32.810Z\",\r\n \"EndTime\": \"2026-06-12T07:26:32.810Z\",\r\n \"CreationTime\": \"2026-06-12T07:26:17.090Z\"\r\n },\r\n {\r\n \"Key\": \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-05T07:34:54.547Z\",\r\n \"EndTime\": \"2026-06-05T07:34:56.163Z\",\r\n \"CreationTime\": \"2026-06-05T07:34:54.547Z\"\r\n },\r\n {\r\n \"Key\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"EndTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"CreationTime\": \"2026-06-04T12:57:51.543Z\"\r\n },\r\n {\r\n \"Key\": \"779be59f-44a2-4447-8105-3306a168813f\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T12:55:26.010Z\",\r\n \"EndTime\": \"2026-06-04T12:55:34.487Z\",\r\n \"CreationTime\": \"2026-06-04T12:55:26.010Z\"\r\n },\r\n {\r\n \"Key\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"EndTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"CreationTime\": \"2026-06-04T10:50:06.750Z\"\r\n },\r\n {\r\n \"Key\": \"b1cab28d-5ab7-42f1-98f3-1920b637ea06\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:42:10.817Z\",\r\n \"EndTime\": \"2026-06-04T10:42:43.897Z\",\r\n \"CreationTime\": \"2026-06-04T10:42:10.817Z\"\r\n },\r\n {\r\n \"Key\": \"8e2c5979-e05f-4e03-b6d1-cda02aa8450c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T15:10:44.610Z\",\r\n \"EndTime\": \"2026-06-03T15:11:17.967Z\",\r\n \"CreationTime\": \"2026-06-03T15:10:44.610Z\"\r\n },\r\n {\r\n \"Key\": \"cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T11:56:44.780Z\",\r\n \"EndTime\": \"2026-06-03T11:56:52.850Z\",\r\n \"CreationTime\": \"2026-06-03T11:56:44.780Z\"\r\n },\r\n {\r\n \"Key\": \"025d0847-ce15-494f-91d6-04be2a19be6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:28:08.133Z\",\r\n \"EndTime\": \"2026-06-03T09:29:10.690Z\",\r\n \"CreationTime\": \"2026-06-03T09:28:08.133Z\"\r\n },\r\n {\r\n \"Key\": \"90058dec-f287-4b4d-8272-1b7cf72e0c0d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:17:53.650Z\",\r\n \"EndTime\": \"2026-06-03T09:18:57.633Z\",\r\n \"CreationTime\": \"2026-06-03T09:17:53.650Z\"\r\n },\r\n {\r\n \"Key\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T09:41:25.187Z\",\r\n \"EndTime\": \"2026-06-02T09:41:53.797Z\",\r\n \"CreationTime\": \"2026-06-02T09:41:25.187Z\"\r\n },\r\n {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\"\r\n },\r\n {\r\n \"Key\": \"68d80da7-719f-44cd-b7d6-94be679f55d4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:52:06.580Z\",\r\n \"EndTime\": \"2026-06-02T06:52:40.293Z\",\r\n \"CreationTime\": \"2026-06-02T06:52:06.580Z\"\r\n },\r\n {\r\n \"Key\": \"3975787c-87d9-4927-bc7c-a19165e06035\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:42:21.737Z\",\r\n \"EndTime\": \"2026-06-02T06:42:56.063Z\",\r\n \"CreationTime\": \"2026-06-02T06:42:21.737Z\"\r\n },\r\n {\r\n \"Key\": \"b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-29T12:28:56.660Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"cb00355c-3835-4c18-b37e-af5c7609c599\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NewBlankTask\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T10:28:07.327Z\",\r\n \"EndTime\": \"2026-05-28T10:29:16.540Z\",\r\n \"CreationTime\": \"2026-05-28T08:48:18.580Z\"\r\n },\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T08:00:50.870Z\",\r\n \"EndTime\": \"2026-05-28T08:01:02.417Z\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 40,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"StartTime\": \"2026-06-12T07:26:49.563Z\",\r\n \"EndTime\": \"2026-06-12T07:27:30.893Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"Info\": \"Element not found\\r\\n\\r\\nin InvoiceEntryUI.xaml\\r\\n at CvClickWithDescriptor \\\"CV Click - 'Button'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at CVScope \\\"CV Screen Scope 'msedge.exe Google'\\\"\\r\\n at Sequence \\\"CV Element Not Found repro\\\"\\r\\n at InvoiceEntryUI \\\"InvoiceEntryUI\\\"\\r\\n\\r\\nUiPath.CV.ElementNotFoundException: Element not found\\r\\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T07:26:49.563Z\",\r\n \"ReleaseName\": \"CV\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21107282,\r\n \"EntryPointPath\": \"InvoiceEntryUI.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T07:27:30.893Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"37e7a8bbd2074db48ebff0ccc7aa85e9\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 66807528\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T07:27:30.476Z\",\r\n \"Message\": \"CV Click - 'Button': Element not found\\r\\nin InvoiceEntryUI.xaml\\r\\n at CvClickWithDescriptor \\\"CV Click - 'Button'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at CVScope \\\"CV Screen Scope 'msedge.exe Google'\\\"\\r\\n at Sequence \\\"CV Element Not Found repro\\\"\\r\\n at InvoiceEntryUI \\\"InvoiceEntryUI\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.CV.ElementNotFoundException\\r\\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T07:27:31.223Z\",\r\n \"JobId\": 66807528\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T07:26:50.023Z\",\r\n \"JobId\": 66807528\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-f493-d5c20a3c0ca5\",\r\n \"TraceId\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"StartTime\": \"2026-06-12T07:26:49.8092821Z\",\r\n \"EndTime\": \"2026-06-12T07:27:30.482966Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.CV.ElementNotFoundException: Element not found\\\\r\\\\n at UiPath.Core.Activities.ObsoleteTaskAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.CV.Activities.CvClickWithDescriptor.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.CV.Activities.CVScope.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\\\r\\\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"c81daadb-3ad3-4681-9fd8-627e57742f4b\",\r\n \"TenantId\": \"3ea4784c-f5f9-4c9f-9cd9-a81c64569726\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"b4a18188-e61d-4786-9f5f-3dc4b8f2c111\",\r\n \"JobKey\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-12T07:27:30.7782589Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CV\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T07:26:49.563Z\",\r\n \"EndTime\": \"2026-06-12T07:27:30.893Z\",\r\n \"CreationTime\": \"2026-06-12T07:26:49.563Z\"\r\n },\r\n {\r\n \"Key\": \"7cb89ff3-dd13-452d-b194-bba867b0e4d9\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"CV\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T07:26:32.810Z\",\r\n \"EndTime\": \"2026-06-12T07:26:32.810Z\",\r\n \"CreationTime\": \"2026-06-12T07:26:17.090Z\"\r\n },\r\n {\r\n \"Key\": \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-05T07:34:54.547Z\",\r\n \"EndTime\": \"2026-06-05T07:34:56.163Z\",\r\n \"CreationTime\": \"2026-06-05T07:34:54.547Z\"\r\n },\r\n {\r\n \"Key\": \"f32f9829-4465-49a2-8e3e-0cc182a020c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T12:57:51.543Z\",\r\n \"EndTime\": \"2026-06-04T12:57:52.790Z\",\r\n \"CreationTime\": \"2026-06-04T12:57:51.543Z\"\r\n },\r\n {\r\n \"Key\": \"779be59f-44a2-4447-8105-3306a168813f\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T12:55:26.010Z\",\r\n \"EndTime\": \"2026-06-04T12:55:34.487Z\",\r\n \"CreationTime\": \"2026-06-04T12:55:26.010Z\"\r\n },\r\n {\r\n \"Key\": \"35c669cb-7f46-4282-b90d-ca7abd52c59f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:50:06.750Z\",\r\n \"EndTime\": \"2026-06-04T10:50:12.563Z\",\r\n \"CreationTime\": \"2026-06-04T10:50:06.750Z\"\r\n },\r\n {\r\n \"Key\": \"b1cab28d-5ab7-42f1-98f3-1920b637ea06\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Legacy\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T10:42:10.817Z\",\r\n \"EndTime\": \"2026-06-04T10:42:43.897Z\",\r\n \"CreationTime\": \"2026-06-04T10:42:10.817Z\"\r\n },\r\n {\r\n \"Key\": \"8e2c5979-e05f-4e03-b6d1-cda02aa8450c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T15:10:44.610Z\",\r\n \"EndTime\": \"2026-06-03T15:11:17.967Z\",\r\n \"CreationTime\": \"2026-06-03T15:10:44.610Z\"\r\n },\r\n {\r\n \"Key\": \"cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T11:56:44.780Z\",\r\n \"EndTime\": \"2026-06-03T11:56:52.850Z\",\r\n \"CreationTime\": \"2026-06-03T11:56:44.780Z\"\r\n },\r\n {\r\n \"Key\": \"025d0847-ce15-494f-91d6-04be2a19be6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:28:08.133Z\",\r\n \"EndTime\": \"2026-06-03T09:29:10.690Z\",\r\n \"CreationTime\": \"2026-06-03T09:28:08.133Z\"\r\n },\r\n {\r\n \"Key\": \"90058dec-f287-4b4d-8272-1b7cf72e0c0d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:17:53.650Z\",\r\n \"EndTime\": \"2026-06-03T09:18:57.633Z\",\r\n \"CreationTime\": \"2026-06-03T09:17:53.650Z\"\r\n },\r\n {\r\n \"Key\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T09:41:25.187Z\",\r\n \"EndTime\": \"2026-06-02T09:41:53.797Z\",\r\n \"CreationTime\": \"2026-06-02T09:41:25.187Z\"\r\n },\r\n {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\"\r\n },\r\n {\r\n \"Key\": \"68d80da7-719f-44cd-b7d6-94be679f55d4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:52:06.580Z\",\r\n \"EndTime\": \"2026-06-02T06:52:40.293Z\",\r\n \"CreationTime\": \"2026-06-02T06:52:06.580Z\"\r\n },\r\n {\r\n \"Key\": \"3975787c-87d9-4927-bc7c-a19165e06035\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:42:21.737Z\",\r\n \"EndTime\": \"2026-06-02T06:42:56.063Z\",\r\n \"CreationTime\": \"2026-06-02T06:42:21.737Z\"\r\n },\r\n {\r\n \"Key\": \"b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-29T12:28:56.660Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"cb00355c-3835-4c18-b37e-af5c7609c599\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NewBlankTask\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T10:28:07.327Z\",\r\n \"EndTime\": \"2026-05-28T10:29:16.540Z\",\r\n \"CreationTime\": \"2026-05-28T08:48:18.580Z\"\r\n },\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T08:00:50.870Z\",\r\n \"EndTime\": \"2026-05-28T08:01:02.417Z\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 40,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key (defb8e05-e36b-4c36-bf11-0b4d08ce6cd1) before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job — the CV process job 37e7a8bb..." + }, + { + "pattern": "or jobs get 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", + "min": 1, + "description": "Agent must fetch the failing job to read the UiPath.CV.ElementNotFoundException 'Element not found' fault from CvClickWithDescriptor (\"CV Click - 'Button'\") inside CVScope (\"CV Screen Scope 'msedge.exe Google'\") in InvoiceEntryUI.xaml." + }, + { + "pattern": "traces", + "min": 1, + "description": "Trace retrieval surfaces the verbatim .NET stack confirming the faulting frame is CvClickWithDescriptor.EndExecute propagating through CVScope.OnFault — pinning the fault to the CV Click find, not scope setup or a downstream action." + }, + { + "pattern": "or jobs logs 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history 37e7a8bb-d207-4db4-8ebf-f0ccc7aa85e9", + "min": 0, + "description": "OPTIONAL — state transitions (Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/task.yaml index 6884bd6392..0755cecf22 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/cv-element-not-found/task.yaml @@ -27,21 +27,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/README.md new file mode 100644 index 0000000000..0e6374f576 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/README.md @@ -0,0 +1,89 @@ +# Connect to Database Failure — ACE OLE DB Provider Not Registered (Bitness Mismatch) + +This scenario reproduces a runtime `Connect to Database` failure caused by a +**provider bitness mismatch**. A **Windows** (64-bit) process reads an Excel +workbook as a database through the ACE OLE DB provider, but the Robot host only +has the **32-bit** Microsoft Access Database Engine installed (bundled with +32-bit Office). Orchestrator returns +`System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is +not registered on the local machine.` + +## What this scenario uncovers + +**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) uses a +well-formed ACE OLE DB connection string +(`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Data\SomeBook.xlsx;...`) with +`ProviderName="System.Data.OleDb"`. `project.json` is **Windows**, so the +process runs 64-bit. The host has only the 32-bit ACE engine, which the 64-bit +process cannot load — so the provider is "not registered" for that process. The +fix is to install the **64-bit** Microsoft Access Database Engine on the Robot +host. The driver bitness must match the **process** bitness, not Office's. + +This maps to: +`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` +— **branch 2 (provider not registered / architecture (bitness) mismatch)**. + +> **Package note:** `Connect to Database` lives in +> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file +> through it is the deliberate "Excel-as-a-database" pattern. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Connect to Database (ACE OLE DB) + Execute Query over `[Sheet1$]`, project type Windows (64-bit) | + +The decisive evidence: + +1. The error: `System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.` pinned to the `Connect to Database` step (`or jobs get` / `or jobs logs`). +2. `Main.xaml` shows a well-formed ACE OLE DB connection string and `ProviderName="System.Data.OleDb"` — ruling out connection-string and provider-init branches. +3. `project.json` `"targetFramework": "Windows"` — the process is 64-bit, so it needs the 64-bit ACE engine. + +## Evidence chain (Orchestrator-only persona) + +The reporting user is off the Robot host and has Orchestrator access only. The +agent diagnoses from Orchestrator evidence plus the workflow source, then hands +back a host-side fix: + +1. `uip or folders list --output json` → find the `Finance Automations` folder key. +2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `ExcelDbReport` job. +3. `uip or jobs get --output json` → read the `InvalidOperationException` in `Info`. +4. `uip or jobs logs --level Error --output json` → confirm the fault is at `Connect to Database`. +5. Read `process/Main.xaml` + `process/project.json` → confirm well-formed ACE connection string + Windows (64-bit) project. + +## How this differs from sibling branches + +| Dimension | branch 1 (conn string) | branch 2 (this) | branch 3 (file lock) | branch 4 (provider init) | +|---|---|---|---|---| +| Connection string well-formed? | no | **yes** | yes | yes | +| `ProviderName` set correctly? | n/a | **yes (`System.Data.OleDb`)** | yes | no / empty → SqlClient | +| Error anchor | `ArgumentException` / initialization string | **"provider is not registered"** | "used by another process" | `Microsoft.Data.SqlClient` type initializer | +| Fix | correct the connection string | **install 64-bit ACE engine** | release the file lock | set `ProviderName` explicitly | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched connect-to-database-failures **branch 2** AND reached the same + root cause as `RESOLUTION.md`: the 64-bit ACE OLE DB provider is not + registered because only the 32-bit engine is installed (bitness mismatch). +- Conclusion must recommend installing the **64-bit** Access Database Engine on + the Robot host (match process bitness, `/quiet` if 32-bit Office blocks it), + and must NOT land on a connection-string typo, file lock, wrong path, SQL + syntax error, or the "install 32-bit ACE to match 32-bit Office" + anti-pattern. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name connect-database-ace-not-registered --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/1a596b4207.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/1a596b4207.json deleted file mode 100644 index 645f8b7a23..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/1a596b4207.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d", - "Name": "Finance Automations", - "Path": "Finance Automations", - "Description": "Production folder for finance back-office processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/537e7696b8.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/537e7696b8.json deleted file mode 100644 index 71a23ab7ba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/537e7696b8.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d", - "State": "Faulted", - "ProcessName": "ExcelDbReport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:14:02.118Z", - "EndTime": "2026-05-21T09:14:03.602Z", - "CreationTime": "2026-05-21T09:14:01.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/78d8ec0f80.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/78d8ec0f80.json deleted file mode 100644 index 0dd7038174..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/78d8ec0f80.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:03.560Z", - "Message": "[Connect to Database] The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:03.588Z", - "Message": "System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/README.md deleted file mode 100644 index 95610d22ab..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/README.md +++ /dev/null @@ -1,88 +0,0 @@ -# Connect to Database Failure — ACE OLE DB Provider Not Registered (Bitness Mismatch) - -This scenario reproduces a runtime `Connect to Database` failure caused by a -**provider bitness mismatch**. A **Windows** (64-bit) process reads an Excel -workbook as a database through the ACE OLE DB provider, but the Robot host only -has the **32-bit** Microsoft Access Database Engine installed (bundled with -32-bit Office). Orchestrator returns -`System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is -not registered on the local machine.` - -## What this scenario uncovers - -**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) uses a -well-formed ACE OLE DB connection string -(`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Data\SomeBook.xlsx;...`) with -`ProviderName="System.Data.OleDb"`. `project.json` is **Windows**, so the -process runs 64-bit. The host has only the 32-bit ACE engine, which the 64-bit -process cannot load — so the provider is "not registered" for that process. The -fix is to install the **64-bit** Microsoft Access Database Engine on the Robot -host. The driver bitness must match the **process** bitness, not Office's. - -This maps to: -`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` -— **branch 2 (provider not registered / architecture (bitness) mismatch)**. - -> **Package note:** `Connect to Database` lives in -> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file -> through it is the deliberate "Excel-as-a-database" pattern. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Connect to Database (ACE OLE DB) + Execute Query over `[Sheet1$]`, project type Windows (64-bit) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The error: `System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.` pinned to the `Connect to Database` step (`or jobs get` / `or jobs logs`). -2. `Main.xaml` shows a well-formed ACE OLE DB connection string and `ProviderName="System.Data.OleDb"` — ruling out connection-string and provider-init branches. -3. `project.json` `"targetFramework": "Windows"` — the process is 64-bit, so it needs the 64-bit ACE engine. - -## Evidence chain (Orchestrator-only persona) - -The reporting user is off the Robot host and has Orchestrator access only. The -agent diagnoses from Orchestrator evidence plus the workflow source, then hands -back a host-side fix: - -1. `uip or folders list --output json` → find the `Finance Automations` folder key. -2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `ExcelDbReport` job. -3. `uip or jobs get --output json` → read the `InvalidOperationException` in `Info`. -4. `uip or jobs logs --level Error --output json` → confirm the fault is at `Connect to Database`. -5. Read `process/Main.xaml` + `process/project.json` → confirm well-formed ACE connection string + Windows (64-bit) project. - -## How this differs from sibling branches - -| Dimension | branch 1 (conn string) | branch 2 (this) | branch 3 (file lock) | branch 4 (provider init) | -|---|---|---|---|---| -| Connection string well-formed? | no | **yes** | yes | yes | -| `ProviderName` set correctly? | n/a | **yes (`System.Data.OleDb`)** | yes | no / empty → SqlClient | -| Error anchor | `ArgumentException` / initialization string | **"provider is not registered"** | "used by another process" | `Microsoft.Data.SqlClient` type initializer | -| Fix | correct the connection string | **install 64-bit ACE engine** | release the file lock | set `ProviderName` explicitly | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched connect-to-database-failures **branch 2** AND reached the same - root cause as `RESOLUTION.md`: the 64-bit ACE OLE DB provider is not - registered because only the 32-bit engine is installed (bitness mismatch). -- Conclusion must recommend installing the **64-bit** Access Database Engine on - the Robot host (match process bitness, `/quiet` if 32-bit Office blocks it), - and must NOT land on a connection-string typo, file lock, wrong path, SQL - syntax error, or the "install 32-bit ACE to match 32-bit Office" - anti-pattern. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name connect-database-ace-not-registered --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/f08d1aaf2b.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/f08d1aaf2b.json deleted file mode 100644 index 9a387fdbbf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/f08d1aaf2b.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d", - "StartTime": "2026-05-21T09:14:02.118Z", - "EndTime": "2026-05-21T09:14:03.602Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c4d5e6f7-8a9b-4c1d-9e2f-3a4b5c6d7e8f", - "Info": "Connect to Database: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\r\n\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.", - "CreationTime": "2026-05-21T09:14:01.880Z", - "ReleaseName": "ExcelDbReport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5041902, - "OrganizationUnitId": 7791624, - "FolderKey": "7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T09:14:03.602Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d", - "Id": 671042118 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/manifest.json deleted file mode 100644 index 65f72bb9da..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Connect to Database scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "1a596b4207.json" - }, - { - "match": "or jobs list --folder-key \"7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d\" --state Faulted --output json", - "file": "537e7696b8.json" - }, - { - "match": "or jobs list --folder-key 7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d --state Faulted --output json", - "file": "537e7696b8.json" - }, - { - "match": "or jobs get \"3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d\" --output json", - "file": "f08d1aaf2b.json" - }, - { - "match": "or jobs get 3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d --output json", - "file": "f08d1aaf2b.json" - }, - { - "match": "or jobs logs \"3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d\" --level Error --output json", - "file": "78d8ec0f80.json" - }, - { - "match": "or jobs logs 3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d --level Error --output json", - "file": "78d8ec0f80.json" - }, - { - "match": "or jobs logs \"3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d\" --output json", - "file": "78d8ec0f80.json" - }, - { - "match": "or jobs logs 3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d --output json", - "file": "78d8ec0f80.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/uip-fixture.json new file mode 100644 index 0000000000..e730bdf498 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ExcelDbReport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:14:02.118Z\",\r\n \"EndTime\": \"2026-05-21T09:14:03.602Z\",\r\n \"CreationTime\": \"2026-05-21T09:14:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.560Z\",\r\n \"Message\": \"[Connect to Database] The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.588Z\",\r\n \"Message\": \"System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d\",\r\n \"StartTime\": \"2026-05-21T09:14:02.118Z\",\r\n \"EndTime\": \"2026-05-21T09:14:03.602Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c4d5e6f7-8a9b-4c1d-9e2f-3a4b5c6d7e8f\",\r\n \"Info\": \"Connect to Database: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\",\r\n \"CreationTime\": \"2026-05-21T09:14:01.880Z\",\r\n \"ReleaseName\": \"ExcelDbReport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5041902,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T09:14:03.602Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d\",\r\n \"Id\": 671042118\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3b8d5c2e-1a4f-4e6c-8b9d-5c7a2e3f4b6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.560Z\",\r\n \"Message\": \"[Connect to Database] The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.588Z\",\r\n \"Message\": \"System.InvalidOperationException: The 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine.\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7c1f2e9a-4b6d-4a8e-9f3c-2d5e6a7b8c9d\",\r\n \"Name\": \"Finance Automations\",\r\n \"Path\": \"Finance Automations\",\r\n \"Description\": \"Production folder for finance back-office processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/task.yaml index a1937310d9..b3cfbfa9fe 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-ace-not-registered/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/README.md new file mode 100644 index 0000000000..2071cf2563 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/README.md @@ -0,0 +1,110 @@ +# Connect to Database Failure — Workbook Held Open by Another Process (File Lock) + +This scenario reproduces a runtime `Connect to Database` failure caused by a +**file lock / sharing violation**. A **Windows** process reads an Excel workbook +as a database through a well-formed ACE OLE DB connection string, but the target +workbook `C:\Finance\Invoices.xlsx` is held open by another process at run time, +so the file-based provider cannot acquire it. Orchestrator returns +`System.IO.IOException: The process cannot access the file +'C:\Finance\Invoices.xlsx' because it is being used by another process.` + +## What this scenario uncovers + +**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) uses a +well-formed ACE OLE DB connection string +(`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Finance\Invoices.xlsx;...`) +with `ProviderName="System.Data.OleDb"`. The connection string, provider, and +downstream SQL are all correct. The workbook is held open by another process — +the Excel UI, an orphaned `EXCEL.EXE` from a prior run, a concurrent job, or a +OneDrive / sync / AV client. File-based providers request a write lock by +default, so the ACE provider cannot acquire the locked file and the connect +throws a sharing violation. The fix is to ensure the workbook is closed (kill +orphan processes) before the run and, for read-only SELECTs, open the connection +read-only (`Mode=Read`). + +This maps to: +`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` +— **branch 3 (file lock / sharing violation)**. + +> **Package note:** `Connect to Database` lives in +> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file +> through it is the deliberate "Excel-as-a-database" pattern. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Connect to Database (ACE OLE DB, well-formed string) + Execute Query over `[Invoices$]`, project type Windows | + +The decisive evidence: + +1. The error: `System.IO.IOException: The process cannot access the file + 'C:\Finance\Invoices.xlsx' because it is being used by another process.` + pinned to the `Connect to Database` step (`or jobs get` / `or jobs logs`), + with frames into `OleDbConnection.Open` and + `UiPath.Database.DatabaseConnection.Connect`. +2. `Main.xaml` shows a **well-formed** ACE OLE DB connection string and the + correct `ProviderName="System.Data.OleDb"` — ruling out the connection-string + (branch 1), provider-not-registered (branch 2), and provider-init (branch 4) + branches. +3. The exception class is `System.IO.IOException` with the "used by another + process" text — a sharing violation, not a configuration error. + +## Evidence chain (Orchestrator-only persona) + +The reporting user has Orchestrator access and the workflow source. The agent +diagnoses from Orchestrator evidence plus the workflow source, then hands back a +host-side fix: + +1. `uip or folders list --output json` → find the `Finance Operations` folder key. +2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `DailyInvoiceLoad` job. +3. `uip or jobs get --output json` → read the `IOException` ("used by another process") in `Info`. +4. `uip or jobs logs --level Error --output json` → confirm the fault is at `Connect to Database`. +5. Read `process/Main.xaml` + `process/project.json` → confirm well-formed ACE connection string + correct provider. + +## How this differs from sibling branches + +| Dimension | branch 1 (conn string) | branch 2 (provider not registered) | branch 3 (this) | branch 4 (provider init) | +|---|---|---|---|---| +| Connection string well-formed? | no | yes | **yes** | yes | +| `ProviderName` set correctly? | n/a | yes (`System.Data.OleDb`) | **yes (`System.Data.OleDb`)** | no / empty → SqlClient | +| Error anchor | `ArgumentException` / initialization string | "provider is not registered" | **`IOException` "used by another process"** | `Microsoft.Data.SqlClient` type initializer | +| Fix | correct the connection string | install 64-bit ACE engine | **release the file lock (close / kill orphan; `Mode=Read`)** | set `ProviderName` explicitly | + +## Synthetic data note + +All fixtures are **synthetic**, hand-authored from the documented branch-3 +signature in the playbook — not captured from a real tenant. The job key, folder +key, host name, and account are mock values (`MOCK-HOST`, `RobotUser1`). Replace +with verbatim captures from a real `.local/investigations/` before treating this +as a hardened regression signal. + +## Success criteria + +The test **scores the conclusion, not the trajectory**: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched connect-to-database-failures **branch 3** AND reached the same + root cause as `RESOLUTION.md`: the workbook is held open by another process + (file lock / sharing violation), and the connection string and provider are + fine. +- Conclusion must recommend closing/releasing the file (kill orphan process) + and/or opening the connection read-only (`Mode=Read`), and must NOT land on a + connection-string typo (branch 1), provider-not-registered / bitness (branch + 2), provider-init / wrong `ProviderName` (branch 4), a SQL syntax error, or a + "file not found / wrong path / missing sheet" conclusion (the file exists — it + is locked). + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name connect-database-file-locked --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/8af2724d70.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/8af2724d70.json deleted file mode 100644 index b3db73a20e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/8af2724d70.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f", - "Name": "Finance Operations", - "Path": "Finance Operations", - "Description": "Production folder for finance back-office processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/README.md deleted file mode 100644 index 1aeeff074c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/README.md +++ /dev/null @@ -1,109 +0,0 @@ -# Connect to Database Failure — Workbook Held Open by Another Process (File Lock) - -This scenario reproduces a runtime `Connect to Database` failure caused by a -**file lock / sharing violation**. A **Windows** process reads an Excel workbook -as a database through a well-formed ACE OLE DB connection string, but the target -workbook `C:\Finance\Invoices.xlsx` is held open by another process at run time, -so the file-based provider cannot acquire it. Orchestrator returns -`System.IO.IOException: The process cannot access the file -'C:\Finance\Invoices.xlsx' because it is being used by another process.` - -## What this scenario uncovers - -**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) uses a -well-formed ACE OLE DB connection string -(`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Finance\Invoices.xlsx;...`) -with `ProviderName="System.Data.OleDb"`. The connection string, provider, and -downstream SQL are all correct. The workbook is held open by another process — -the Excel UI, an orphaned `EXCEL.EXE` from a prior run, a concurrent job, or a -OneDrive / sync / AV client. File-based providers request a write lock by -default, so the ACE provider cannot acquire the locked file and the connect -throws a sharing violation. The fix is to ensure the workbook is closed (kill -orphan processes) before the run and, for read-only SELECTs, open the connection -read-only (`Mode=Read`). - -This maps to: -`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` -— **branch 3 (file lock / sharing violation)**. - -> **Package note:** `Connect to Database` lives in -> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file -> through it is the deliberate "Excel-as-a-database" pattern. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Connect to Database (ACE OLE DB, well-formed string) + Execute Query over `[Invoices$]`, project type Windows | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-3 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The error: `System.IO.IOException: The process cannot access the file - 'C:\Finance\Invoices.xlsx' because it is being used by another process.` - pinned to the `Connect to Database` step (`or jobs get` / `or jobs logs`), - with frames into `OleDbConnection.Open` and - `UiPath.Database.DatabaseConnection.Connect`. -2. `Main.xaml` shows a **well-formed** ACE OLE DB connection string and the - correct `ProviderName="System.Data.OleDb"` — ruling out the connection-string - (branch 1), provider-not-registered (branch 2), and provider-init (branch 4) - branches. -3. The exception class is `System.IO.IOException` with the "used by another - process" text — a sharing violation, not a configuration error. - -## Evidence chain (Orchestrator-only persona) - -The reporting user has Orchestrator access and the workflow source. The agent -diagnoses from Orchestrator evidence plus the workflow source, then hands back a -host-side fix: - -1. `uip or folders list --output json` → find the `Finance Operations` folder key. -2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `DailyInvoiceLoad` job. -3. `uip or jobs get --output json` → read the `IOException` ("used by another process") in `Info`. -4. `uip or jobs logs --level Error --output json` → confirm the fault is at `Connect to Database`. -5. Read `process/Main.xaml` + `process/project.json` → confirm well-formed ACE connection string + correct provider. - -## How this differs from sibling branches - -| Dimension | branch 1 (conn string) | branch 2 (provider not registered) | branch 3 (this) | branch 4 (provider init) | -|---|---|---|---|---| -| Connection string well-formed? | no | yes | **yes** | yes | -| `ProviderName` set correctly? | n/a | yes (`System.Data.OleDb`) | **yes (`System.Data.OleDb`)** | no / empty → SqlClient | -| Error anchor | `ArgumentException` / initialization string | "provider is not registered" | **`IOException` "used by another process"** | `Microsoft.Data.SqlClient` type initializer | -| Fix | correct the connection string | install 64-bit ACE engine | **release the file lock (close / kill orphan; `Mode=Read`)** | set `ProviderName` explicitly | - -## Synthetic data note - -All fixtures are **synthetic**, hand-authored from the documented branch-3 -signature in the playbook — not captured from a real tenant. The job key, folder -key, host name, and account are mock values (`MOCK-HOST`, `RobotUser1`). Replace -with verbatim captures from a real `.local/investigations/` before treating this -as a hardened regression signal. - -## Success criteria - -The test **scores the conclusion, not the trajectory**: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched connect-to-database-failures **branch 3** AND reached the same - root cause as `RESOLUTION.md`: the workbook is held open by another process - (file lock / sharing violation), and the connection string and provider are - fine. -- Conclusion must recommend closing/releasing the file (kill orphan process) - and/or opening the connection read-only (`Mode=Read`), and must NOT land on a - connection-string typo (branch 1), provider-not-registered / bitness (branch - 2), provider-init / wrong `ProviderName` (branch 4), a SQL syntax error, or a - "file not found / wrong path / missing sheet" conclusion (the file exists — it - is locked). - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name connect-database-file-locked --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/abbcc9f48d.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/abbcc9f48d.json deleted file mode 100644 index 3e440d9cb3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/abbcc9f48d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80", - "StartTime": "2026-06-01T05:40:01.204Z", - "EndTime": "2026-06-01T05:40:02.337Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "a7b8c9d0-1e2f-4a3b-9c4d-5e6f7a8b9c0d", - "Info": "Connect to Database: A database error occurred.\r\n\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.IOException: The process cannot access the file 'C:\\Finance\\Invoices.xlsx' because it is being used by another process.\r\n at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)\r\n at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)\r\n at System.Data.OleDb.OleDbConnection.Open()\r\n at UiPath.Database.DatabaseConnection.Connect(String connString, String provider, Int32 timeout)", - "CreationTime": "2026-06-01T05:40:00.940Z", - "ReleaseName": "DailyInvoiceLoad", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5042118, - "OrganizationUnitId": 7791630, - "FolderKey": "e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-01T05:40:02.337Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80", - "Id": 671048204 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/c0fb23d640.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/c0fb23d640.json deleted file mode 100644 index 9c2d47431b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/c0fb23d640.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80", - "State": "Faulted", - "ProcessName": "DailyInvoiceLoad", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-01T05:40:01.204Z", - "EndTime": "2026-06-01T05:40:02.337Z", - "CreationTime": "2026-06-01T05:40:00.940Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/c4dbd28372.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/c4dbd28372.json deleted file mode 100644 index 5d377b13fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/c4dbd28372.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-01T05:40:02.290Z", - "Message": "[Connect to Database] The process cannot access the file 'C:\\Finance\\Invoices.xlsx' because it is being used by another process." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-01T05:40:02.318Z", - "Message": "System.IO.IOException: The process cannot access the file 'C:\\Finance\\Invoices.xlsx' because it is being used by another process.\r\n at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)\r\n at System.Data.OleDb.OleDbConnection.Open()\r\n at UiPath.Database.DatabaseConnection.Connect(String connString, String provider, Int32 timeout)\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/manifest.json deleted file mode 100644 index 401f466851..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Connect to Database scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "8af2724d70.json" - }, - { - "match": "or jobs list --folder-key \"e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f\" --state Faulted --output json", - "file": "c0fb23d640.json" - }, - { - "match": "or jobs list --folder-key e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f --state Faulted --output json", - "file": "c0fb23d640.json" - }, - { - "match": "or jobs get \"f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80\" --output json", - "file": "abbcc9f48d.json" - }, - { - "match": "or jobs get f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80 --output json", - "file": "abbcc9f48d.json" - }, - { - "match": "or jobs logs \"f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80\" --level Error --output json", - "file": "c4dbd28372.json" - }, - { - "match": "or jobs logs f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80 --level Error --output json", - "file": "c4dbd28372.json" - }, - { - "match": "or jobs logs \"f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80\" --output json", - "file": "c4dbd28372.json" - }, - { - "match": "or jobs logs f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80 --output json", - "file": "c4dbd28372.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/uip-fixture.json new file mode 100644 index 0000000000..68fa8509cc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyInvoiceLoad\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-01T05:40:01.204Z\",\r\n \"EndTime\": \"2026-06-01T05:40:02.337Z\",\r\n \"CreationTime\": \"2026-06-01T05:40:00.940Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T05:40:02.290Z\",\r\n \"Message\": \"[Connect to Database] The process cannot access the file 'C:\\\\Finance\\\\Invoices.xlsx' because it is being used by another process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T05:40:02.318Z\",\r\n \"Message\": \"System.IO.IOException: The process cannot access the file 'C:\\\\Finance\\\\Invoices.xlsx' because it is being used by another process.\\r\\n at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)\\r\\n at System.Data.OleDb.OleDbConnection.Open()\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connString, String provider, Int32 timeout)\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80\",\r\n \"StartTime\": \"2026-06-01T05:40:01.204Z\",\r\n \"EndTime\": \"2026-06-01T05:40:02.337Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"a7b8c9d0-1e2f-4a3b-9c4d-5e6f7a8b9c0d\",\r\n \"Info\": \"Connect to Database: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.IOException: The process cannot access the file 'C:\\\\Finance\\\\Invoices.xlsx' because it is being used by another process.\\r\\n at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)\\r\\n at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)\\r\\n at System.Data.OleDb.OleDbConnection.Open()\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connString, String provider, Int32 timeout)\",\r\n \"CreationTime\": \"2026-06-01T05:40:00.940Z\",\r\n \"ReleaseName\": \"DailyInvoiceLoad\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5042118,\r\n \"OrganizationUnitId\": 7791630,\r\n \"FolderKey\": \"e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-01T05:40:02.337Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80\",\r\n \"Id\": 671048204\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f4d5e6a7-8b9c-4d0e-9f1a-3b4c5d6e7f80", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T05:40:02.290Z\",\r\n \"Message\": \"[Connect to Database] The process cannot access the file 'C:\\\\Finance\\\\Invoices.xlsx' because it is being used by another process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T05:40:02.318Z\",\r\n \"Message\": \"System.IO.IOException: The process cannot access the file 'C:\\\\Finance\\\\Invoices.xlsx' because it is being used by another process.\\r\\n at System.Data.OleDb.OleDbConnectionInternal..ctor(OleDbConnectionString constr, OleDbConnection connection)\\r\\n at System.Data.OleDb.OleDbConnection.Open()\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connString, String provider, Int32 timeout)\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e3c4d5f6-7a8b-4c9d-8e0f-2a3b4c5d6e7f\",\r\n \"Name\": \"Finance Operations\",\r\n \"Path\": \"Finance Operations\",\r\n \"Description\": \"Production folder for finance back-office processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/task.yaml index d309865051..5836b6d12c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-file-locked/task.yaml @@ -28,22 +28,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/README.md new file mode 100644 index 0000000000..4801094910 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/README.md @@ -0,0 +1,95 @@ +# Connect to Database Failure — Malformed Connection String (Unquoted Extended Properties) + +This scenario reproduces a runtime `Connect to Database` failure caused by a +**malformed connection string**. A **Windows** process reads an Excel workbook +as a database through the ACE OLE DB provider, but the `Extended Properties` +value in the connection string is **not quoted**, so the ADO.NET +connection-string parser cannot parse it. Orchestrator returns +`System.ArgumentException: Format of the initialization string does not conform +to specification starting at index 47.` + +## What this scenario uncovers + +**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) uses +`ProviderName="System.Data.OleDb"` and the connection string +`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Finance\Recon.xlsx;Extended +Properties=Excel 12.0 Xml;HDR=YES;`. A connection string is parsed as +`keyword=value;` pairs. The `Extended Properties` value itself contains `;`, so +it must be quoted (`Extended Properties="Excel 12.0 Xml;HDR=YES;"`). Without the +quotes, the parser reads the embedded `;` as a key/value separator and rejects +the string before the provider, the file, or the query are ever touched. The +fix is to **quote** the `Extended Properties` value. + +This maps to: +`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` +— **branch 1 (malformed / wrong connection string)**. + +> **Package note:** `Connect to Database` lives in +> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file +> through it is the deliberate "Excel-as-a-database" pattern. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Connect to Database (ACE OLE DB) with an **unquoted** `Extended Properties` + a downstream Execute Query over `[Sheet1$]`, project type Windows | + +The decisive evidence: + +1. The error: `System.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.` pinned to the `Connect to Database` step, with frames in `System.Data.Common.DbConnectionOptions` and `UiPath.Database.DatabaseConnection.Connect` (`or jobs get` / `or jobs logs`). +2. `Main.xaml` shows the **unquoted** `Extended Properties=Excel 12.0 Xml;HDR=YES;` in the ACE OLE DB connection string — the embedded `;` breaks the `keyword=value;` layout. +3. `ProviderName="System.Data.OleDb"` is correct (rules out provider-init / wrong-provider branch 4); the failure is a connection-string *format* error, not a driver, file, or query error. + +## Evidence chain + +The agent diagnoses from Orchestrator evidence plus the workflow source: + +1. `uip or folders list --output json` → find the `Finance Operations` folder key. +2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `MonthlyReconImport` job. +3. `uip or jobs get --output json` → read the `ArgumentException` / "initialization string" message in `Info`. +4. `uip or jobs logs --level Error --output json` → confirm the fault is at `Connect to Database`, in the connection-string parser. +5. Read `process/Main.xaml` → confirm the unquoted `Extended Properties` in the ConnectionString. + +## How this differs from sibling branches + +| Dimension | branch 1 (this) | branch 2 (provider not registered) | branch 3 (file lock) | branch 4 (provider init) | +|---|---|---|---|---| +| Connection string well-formed? | **no (unquoted Extended Properties)** | yes | yes | yes | +| `ProviderName` set correctly? | yes (`System.Data.OleDb`) | yes | yes | no / empty → SqlClient | +| Error anchor | **`ArgumentException` / "initialization string"** | "provider is not registered" | "used by another process" | `Microsoft.Data.SqlClient` type initializer | +| Fix | **quote the Extended Properties value** | install 64-bit ACE engine | release the file lock | set `ProviderName` explicitly | + +## Synthetic fixtures + +branch-1 signature, not captured from a real `.local/investigations/` session. +Replace them with verbatim captures before treating this as a regression +signal (see `../_shared/scripts/generate_scenario.py`). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched connect-to-database-failures **branch 1** AND reached the same + root cause as `RESOLUTION.md`: the malformed connection string — the + unquoted `Extended Properties` value — causes the parser to throw "Format of + the initialization string does not conform to specification". +- Conclusion must recommend **quoting** the `Extended Properties` value + (`Extended Properties="Excel 12.0 Xml;HDR=YES;"`), and must NOT land on + provider-not-registered / bitness (branch 2), a file lock (branch 3), + provider-init / wrong `ProviderName` (branch 4), a SQL syntax error, a missing + file/sheet, or a database outage. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name connect-database-malformed-connstring --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/51c0164e62.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/51c0164e62.json deleted file mode 100644 index cdd5d47e16..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/51c0164e62.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "State": "Faulted", - "ProcessName": "MonthlyReconImport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-01T07:05:00.142Z", - "EndTime": "2026-06-01T07:05:00.948Z", - "CreationTime": "2026-06-01T07:04:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/739fa596f7.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/739fa596f7.json deleted file mode 100644 index 5dd6e62091..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/739fa596f7.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Name": "Finance Operations", - "Path": "Finance Operations", - "Description": "Production folder for finance back-office processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/README.md deleted file mode 100644 index a6c76ebafd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/README.md +++ /dev/null @@ -1,95 +0,0 @@ -# Connect to Database Failure — Malformed Connection String (Unquoted Extended Properties) - -This scenario reproduces a runtime `Connect to Database` failure caused by a -**malformed connection string**. A **Windows** process reads an Excel workbook -as a database through the ACE OLE DB provider, but the `Extended Properties` -value in the connection string is **not quoted**, so the ADO.NET -connection-string parser cannot parse it. Orchestrator returns -`System.ArgumentException: Format of the initialization string does not conform -to specification starting at index 47.` - -## What this scenario uncovers - -**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) uses -`ProviderName="System.Data.OleDb"` and the connection string -`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Finance\Recon.xlsx;Extended -Properties=Excel 12.0 Xml;HDR=YES;`. A connection string is parsed as -`keyword=value;` pairs. The `Extended Properties` value itself contains `;`, so -it must be quoted (`Extended Properties="Excel 12.0 Xml;HDR=YES;"`). Without the -quotes, the parser reads the embedded `;` as a key/value separator and rejects -the string before the provider, the file, or the query are ever touched. The -fix is to **quote** the `Extended Properties` value. - -This maps to: -`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` -— **branch 1 (malformed / wrong connection string)**. - -> **Package note:** `Connect to Database` lives in -> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file -> through it is the deliberate "Excel-as-a-database" pattern. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Connect to Database (ACE OLE DB) with an **unquoted** `Extended Properties` + a downstream Execute Query over `[Sheet1$]`, project type Windows | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-1 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The error: `System.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.` pinned to the `Connect to Database` step, with frames in `System.Data.Common.DbConnectionOptions` and `UiPath.Database.DatabaseConnection.Connect` (`or jobs get` / `or jobs logs`). -2. `Main.xaml` shows the **unquoted** `Extended Properties=Excel 12.0 Xml;HDR=YES;` in the ACE OLE DB connection string — the embedded `;` breaks the `keyword=value;` layout. -3. `ProviderName="System.Data.OleDb"` is correct (rules out provider-init / wrong-provider branch 4); the failure is a connection-string *format* error, not a driver, file, or query error. - -## Evidence chain - -The agent diagnoses from Orchestrator evidence plus the workflow source: - -1. `uip or folders list --output json` → find the `Finance Operations` folder key. -2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `MonthlyReconImport` job. -3. `uip or jobs get --output json` → read the `ArgumentException` / "initialization string" message in `Info`. -4. `uip or jobs logs --level Error --output json` → confirm the fault is at `Connect to Database`, in the connection-string parser. -5. Read `process/Main.xaml` → confirm the unquoted `Extended Properties` in the ConnectionString. - -## How this differs from sibling branches - -| Dimension | branch 1 (this) | branch 2 (provider not registered) | branch 3 (file lock) | branch 4 (provider init) | -|---|---|---|---|---| -| Connection string well-formed? | **no (unquoted Extended Properties)** | yes | yes | yes | -| `ProviderName` set correctly? | yes (`System.Data.OleDb`) | yes | yes | no / empty → SqlClient | -| Error anchor | **`ArgumentException` / "initialization string"** | "provider is not registered" | "used by another process" | `Microsoft.Data.SqlClient` type initializer | -| Fix | **quote the Extended Properties value** | install 64-bit ACE engine | release the file lock | set `ProviderName` explicitly | - -## Synthetic fixtures - -The `fixtures/mocks/responses/*.json` are **hand-authored** from the documented -branch-1 signature, not captured from a real `.local/investigations/` session. -Replace them with verbatim captures before treating this as a regression -signal (see `../_shared/scripts/generate_scenario.py`). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched connect-to-database-failures **branch 1** AND reached the same - root cause as `RESOLUTION.md`: the malformed connection string — the - unquoted `Extended Properties` value — causes the parser to throw "Format of - the initialization string does not conform to specification". -- Conclusion must recommend **quoting** the `Extended Properties` value - (`Extended Properties="Excel 12.0 Xml;HDR=YES;"`), and must NOT land on - provider-not-registered / bitness (branch 2), a file lock (branch 3), - provider-init / wrong `ProviderName` (branch 4), a SQL syntax error, a missing - file/sheet, or a database outage. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name connect-database-malformed-connstring --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/c0f27643d7.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/c0f27643d7.json deleted file mode 100644 index 6be4154bce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/c0f27643d7.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-01T07:05:00.910Z", - "Message": "[Connect to Database] A database error occurred. Format of the initialization string does not conform to specification starting at index 47." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-01T07:05:00.938Z", - "Message": "System.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.\r\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\r\n at System.Data.Common.DbConnectionOptions..ctor(String connectionString, Dictionary`2 synonyms, Boolean useOdbcRules)\r\n at System.Data.OleDb.OleDbConnectionString..ctor(String connectionString, Boolean validate)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String providerName)\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/ee20a281f1.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/ee20a281f1.json deleted file mode 100644 index 9fa2590e22..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/ee20a281f1.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "StartTime": "2026-06-01T07:05:00.142Z", - "EndTime": "2026-06-01T07:05:00.948Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "e6f7a8b9-0c1d-4e2f-8a3b-4c5d6e7f8a9b", - "Info": "Connect to Database: A database error occurred.\r\n\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.\r\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\r\n at System.Data.Common.DbConnectionOptions.ParseInternal(Dictionary`2 parsetable, String connectionString, Boolean buildChain, Dictionary`2 synonyms, Boolean firstKey)\r\n at System.Data.Common.DbConnectionOptions..ctor(String connectionString, Dictionary`2 synonyms, Boolean useOdbcRules)\r\n at System.Data.OleDb.OleDbConnectionString..ctor(String connectionString, Boolean validate)\r\n at System.Data.OleDb.OleDbConnectionFactory.CreateConnectionOptions(String connectionString, DbConnectionOptions previous)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String providerName)", - "CreationTime": "2026-06-01T07:04:59.910Z", - "ReleaseName": "MonthlyReconImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5041902, - "OrganizationUnitId": 7791624, - "FolderKey": "c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-01T07:05:00.948Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Id": 671059910 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/manifest.json deleted file mode 100644 index 2e53c1cc49..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Connect to Database scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "739fa596f7.json" - }, - { - "match": "or jobs list --folder-key \"c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d\" --state Faulted --output json", - "file": "51c0164e62.json" - }, - { - "match": "or jobs list --folder-key c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d --state Faulted --output json", - "file": "51c0164e62.json" - }, - { - "match": "or jobs get \"d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e\" --output json", - "file": "ee20a281f1.json" - }, - { - "match": "or jobs get d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e --output json", - "file": "ee20a281f1.json" - }, - { - "match": "or jobs logs \"d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e\" --level Error --output json", - "file": "c0f27643d7.json" - }, - { - "match": "or jobs logs d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e --level Error --output json", - "file": "c0f27643d7.json" - }, - { - "match": "or jobs logs \"d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e\" --output json", - "file": "c0f27643d7.json" - }, - { - "match": "or jobs logs d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e --output json", - "file": "c0f27643d7.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/uip-fixture.json new file mode 100644 index 0000000000..ab10fe2446 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MonthlyReconImport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-01T07:05:00.142Z\",\r\n \"EndTime\": \"2026-06-01T07:05:00.948Z\",\r\n \"CreationTime\": \"2026-06-01T07:04:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T07:05:00.910Z\",\r\n \"Message\": \"[Connect to Database] A database error occurred. Format of the initialization string does not conform to specification starting at index 47.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T07:05:00.938Z\",\r\n \"Message\": \"System.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.\\r\\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\\r\\n at System.Data.Common.DbConnectionOptions..ctor(String connectionString, Dictionary`2 synonyms, Boolean useOdbcRules)\\r\\n at System.Data.OleDb.OleDbConnectionString..ctor(String connectionString, Boolean validate)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String providerName)\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"StartTime\": \"2026-06-01T07:05:00.142Z\",\r\n \"EndTime\": \"2026-06-01T07:05:00.948Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"e6f7a8b9-0c1d-4e2f-8a3b-4c5d6e7f8a9b\",\r\n \"Info\": \"Connect to Database: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.\\r\\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\\r\\n at System.Data.Common.DbConnectionOptions.ParseInternal(Dictionary`2 parsetable, String connectionString, Boolean buildChain, Dictionary`2 synonyms, Boolean firstKey)\\r\\n at System.Data.Common.DbConnectionOptions..ctor(String connectionString, Dictionary`2 synonyms, Boolean useOdbcRules)\\r\\n at System.Data.OleDb.OleDbConnectionString..ctor(String connectionString, Boolean validate)\\r\\n at System.Data.OleDb.OleDbConnectionFactory.CreateConnectionOptions(String connectionString, DbConnectionOptions previous)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String providerName)\",\r\n \"CreationTime\": \"2026-06-01T07:04:59.910Z\",\r\n \"ReleaseName\": \"MonthlyReconImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5041902,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-01T07:05:00.948Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Id\": 671059910\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d2b3c4e5-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T07:05:00.910Z\",\r\n \"Message\": \"[Connect to Database] A database error occurred. Format of the initialization string does not conform to specification starting at index 47.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T07:05:00.938Z\",\r\n \"Message\": \"System.ArgumentException: Format of the initialization string does not conform to specification starting at index 47.\\r\\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\\r\\n at System.Data.Common.DbConnectionOptions..ctor(String connectionString, Dictionary`2 synonyms, Boolean useOdbcRules)\\r\\n at System.Data.OleDb.OleDbConnectionString..ctor(String connectionString, Boolean validate)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String providerName)\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c1a2b3d4-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Finance Operations\",\r\n \"Path\": \"Finance Operations\",\r\n \"Description\": \"Production folder for finance back-office processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/task.yaml index de36076b3e..7fb7ca000f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-malformed-connstring/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/README.md new file mode 100644 index 0000000000..01e6e1c53e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/README.md @@ -0,0 +1,101 @@ +# Connect to Database Failure — Provider Init / Empty ProviderName After Windows Migration + +This scenario reproduces a runtime `Connect to Database` failure caused by an +**empty `ProviderName`** on a project migrated from **Windows - Legacy** to +**Windows** (.NET). The `LedgerExportXlsx` process reads an Excel workbook as a +database through an ACE OLE DB connection string, but with no provider set the +modern runtime resolves to **SqlClient** and faults with +`System.TypeInitializationException: The type initializer for +'Microsoft.Data.SqlClient.SqlConnection' threw an exception.` — even though the +intended target is Excel. + +## What this scenario uncovers + +**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) has +`ProviderName=""` (empty) and a well-formed ACE OLE DB connection string +(`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Finance\Ledger.xlsx;...`). +`project.json` is **Windows**, so on the modern .NET runtime the empty provider +resolves to SqlClient; the activity initializes `Microsoft.Data.SqlClient` and +the `SqlConnection` type initializer throws. The fix is to set `ProviderName` +explicitly to `System.Data.OleDb` — the source is Excel/Access, **not** SQL +Server, so switching to `Microsoft.Data.SqlClient` is the wrong fix. + +This maps to: +`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` +— **branch 4 (provider init failure after migration / wrong `ProviderName`)**. + +> **Package note:** `Connect to Database` lives in +> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file +> through it is the deliberate "Excel-as-a-database" pattern. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Connect to Database (empty ProviderName, ACE OLE DB string) + Execute Query over `[Ledger$]`, project type Windows | + +The decisive evidence: + +1. The error: + `System.TypeInitializationException: The type initializer for + 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.` pinned to the + `Connect to Database` step (`or jobs get` / `or jobs logs`), with frames into + `Microsoft.Data.SqlClient` and `UiPath.Database.DatabaseConnection.Connect`. +2. `Main.xaml` shows `ProviderName=""` (empty) alongside a well-formed ACE OLE + DB connection string — ruling out the connection-string and bitness branches. +3. `project.json` `"targetFramework": "Windows"` — the project was migrated off + Windows - Legacy, where the empty provider was tolerated. +4. The job logs note that the runtime resolved the empty `ProviderName` to the + SqlClient provider while the connection string is an ACE/Excel OLE DB string. + +## Evidence chain + +1. `uip or folders list --output json` → find the `Finance Operations` folder key. +2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `LedgerExportXlsx` job. +3. `uip or jobs get --output json` → read the `TypeInitializationException` in `Info`. +4. `uip or jobs logs --level Error --output json` → confirm the empty-provider-resolves-to-SqlClient note. +5. Read `process/Main.xaml` + `process/project.json` → confirm empty `ProviderName` + ACE connection string + Windows (migrated) project. + +## How this differs from sibling branches + +| Dimension | branch 1 (conn string) | branch 2 (bitness) | branch 3 (file lock) | branch 4 (this) | +|---|---|---|---|---| +| Connection string well-formed? | no | yes | yes | **yes** | +| `ProviderName` set correctly? | n/a | yes (`System.Data.OleDb`) | yes | **no — empty → SqlClient** | +| Error anchor | `ArgumentException` / initialization string | "provider is not registered" | "used by another process" | **`Microsoft.Data.SqlClient` type initializer** | +| Fix | correct the connection string | install 64-bit ACE engine | release the file lock | **set `ProviderName="System.Data.OleDb"`** | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched connect-to-database-failures **branch 4** AND reached the same + root cause as `RESOLUTION.md`: the empty `ProviderName` on the migrated + Windows project resolves to SqlClient, throwing the + `Microsoft.Data.SqlClient` type-initializer exception against an Excel/ACE OLE + DB source. +- Conclusion must recommend setting `ProviderName` explicitly to + `System.Data.OleDb`, and must NOT land on a connection-string typo, a bitness + / "provider not registered" mismatch, a file lock, a SQL syntax error, or the + "switch to Microsoft.Data.SqlClient" anti-pattern (the target is Excel/OLE + DB). + +> **Synthetic note:** All fixtures are hand-authored from the documented +> branch-4 signature, not captured from a live tenant. Replace with verbatim +> `.local/investigations/` captures before treating this as a hard regression +> signal. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name connect-database-provider-init --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/4561db8e13.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/4561db8e13.json deleted file mode 100644 index 587f0b3b2c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/4561db8e13.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091", - "Name": "Finance Operations", - "Path": "Finance Operations", - "Description": "Production folder for finance operations processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c7d8e9fa-0b1c-4d2e-8f3a-6b7c8d9e0f12", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/7a748858c9.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/7a748858c9.json deleted file mode 100644 index 808ce19c1c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/7a748858c9.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102", - "StartTime": "2026-06-01T09:50:00.214Z", - "EndTime": "2026-06-01T09:50:01.331Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "a725110e-db4d-441f-9c86-8095106f6562", - "Info": "Connect to Database: A database error occurred.\r\n\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\r\n at Microsoft.Data.SqlClient.SqlConnection..ctor(String connectionString)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider)", - "CreationTime": "2026-06-01T09:49:59.902Z", - "ReleaseName": "LedgerExportXlsx", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5052114, - "OrganizationUnitId": 7791624, - "FolderKey": "a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-01T09:50:01.331Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102", - "Id": 671050214 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/README.md deleted file mode 100644 index 1c6c0324fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/README.md +++ /dev/null @@ -1,100 +0,0 @@ -# Connect to Database Failure — Provider Init / Empty ProviderName After Windows Migration - -This scenario reproduces a runtime `Connect to Database` failure caused by an -**empty `ProviderName`** on a project migrated from **Windows - Legacy** to -**Windows** (.NET). The `LedgerExportXlsx` process reads an Excel workbook as a -database through an ACE OLE DB connection string, but with no provider set the -modern runtime resolves to **SqlClient** and faults with -`System.TypeInitializationException: The type initializer for -'Microsoft.Data.SqlClient.SqlConnection' threw an exception.` — even though the -intended target is Excel. - -## What this scenario uncovers - -**Root Cause:** The `Connect to Database` activity (`DatabaseConnect`) has -`ProviderName=""` (empty) and a well-formed ACE OLE DB connection string -(`Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\Finance\Ledger.xlsx;...`). -`project.json` is **Windows**, so on the modern .NET runtime the empty provider -resolves to SqlClient; the activity initializes `Microsoft.Data.SqlClient` and -the `SqlConnection` type initializer throws. The fix is to set `ProviderName` -explicitly to `System.Data.OleDb` — the source is Excel/Access, **not** SQL -Server, so switching to `Microsoft.Data.SqlClient` is the wrong fix. - -This maps to: -`references/activity-packages/database-activities/playbooks/connect-to-database-failures.md` -— **branch 4 (provider init failure after migration / wrong `ProviderName`)**. - -> **Package note:** `Connect to Database` lives in -> `UiPath.Database.Activities`, NOT the Excel package. Reading an Excel file -> through it is the deliberate "Excel-as-a-database" pattern. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Connect to Database (empty ProviderName, ACE OLE DB string) + Execute Query over `[Ledger$]`, project type Windows | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-4 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The error: - `System.TypeInitializationException: The type initializer for - 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.` pinned to the - `Connect to Database` step (`or jobs get` / `or jobs logs`), with frames into - `Microsoft.Data.SqlClient` and `UiPath.Database.DatabaseConnection.Connect`. -2. `Main.xaml` shows `ProviderName=""` (empty) alongside a well-formed ACE OLE - DB connection string — ruling out the connection-string and bitness branches. -3. `project.json` `"targetFramework": "Windows"` — the project was migrated off - Windows - Legacy, where the empty provider was tolerated. -4. The job logs note that the runtime resolved the empty `ProviderName` to the - SqlClient provider while the connection string is an ACE/Excel OLE DB string. - -## Evidence chain - -1. `uip or folders list --output json` → find the `Finance Operations` folder key. -2. `uip or jobs list --folder-key --state Faulted --output json` → find the faulted `LedgerExportXlsx` job. -3. `uip or jobs get --output json` → read the `TypeInitializationException` in `Info`. -4. `uip or jobs logs --level Error --output json` → confirm the empty-provider-resolves-to-SqlClient note. -5. Read `process/Main.xaml` + `process/project.json` → confirm empty `ProviderName` + ACE connection string + Windows (migrated) project. - -## How this differs from sibling branches - -| Dimension | branch 1 (conn string) | branch 2 (bitness) | branch 3 (file lock) | branch 4 (this) | -|---|---|---|---|---| -| Connection string well-formed? | no | yes | yes | **yes** | -| `ProviderName` set correctly? | n/a | yes (`System.Data.OleDb`) | yes | **no — empty → SqlClient** | -| Error anchor | `ArgumentException` / initialization string | "provider is not registered" | "used by another process" | **`Microsoft.Data.SqlClient` type initializer** | -| Fix | correct the connection string | install 64-bit ACE engine | release the file lock | **set `ProviderName="System.Data.OleDb"`** | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched connect-to-database-failures **branch 4** AND reached the same - root cause as `RESOLUTION.md`: the empty `ProviderName` on the migrated - Windows project resolves to SqlClient, throwing the - `Microsoft.Data.SqlClient` type-initializer exception against an Excel/ACE OLE - DB source. -- Conclusion must recommend setting `ProviderName` explicitly to - `System.Data.OleDb`, and must NOT land on a connection-string typo, a bitness - / "provider not registered" mismatch, a file lock, a SQL syntax error, or the - "switch to Microsoft.Data.SqlClient" anti-pattern (the target is Excel/OLE - DB). - -> **Synthetic note:** All fixtures are hand-authored from the documented -> branch-4 signature, not captured from a live tenant. Replace with verbatim -> `.local/investigations/` captures before treating this as a hard regression -> signal. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name connect-database-provider-init --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/b501343dd9.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/b501343dd9.json deleted file mode 100644 index aefde90a57..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/b501343dd9.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102", - "State": "Faulted", - "ProcessName": "LedgerExportXlsx", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-01T09:50:00.214Z", - "EndTime": "2026-06-01T09:50:01.331Z", - "CreationTime": "2026-06-01T09:49:59.902Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/eb9f9ed47f.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/eb9f9ed47f.json deleted file mode 100644 index 2f9542f669..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/eb9f9ed47f.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-01T09:50:01.288Z", - "Message": "[Connect to Database] A database error occurred. The runtime resolved an empty ProviderName to the SqlClient provider (Microsoft.Data.SqlClient), but the connection string is an ACE/Excel OLE DB string (Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\\Finance\\Ledger.xlsx)." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-01T09:50:01.312Z", - "Message": "System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\r\n at Microsoft.Data.SqlClient.SqlConnection..ctor(String connectionString)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider)\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/manifest.json deleted file mode 100644 index 16106ab654..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Connect to Database scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "4561db8e13.json" - }, - { - "match": "or jobs list --folder-key \"a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091\" --state Faulted --output json", - "file": "b501343dd9.json" - }, - { - "match": "or jobs list --folder-key a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091 --state Faulted --output json", - "file": "b501343dd9.json" - }, - { - "match": "or jobs get \"b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102\" --output json", - "file": "7a748858c9.json" - }, - { - "match": "or jobs get b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102 --output json", - "file": "7a748858c9.json" - }, - { - "match": "or jobs logs \"b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102\" --level Error --output json", - "file": "eb9f9ed47f.json" - }, - { - "match": "or jobs logs b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102 --level Error --output json", - "file": "eb9f9ed47f.json" - }, - { - "match": "or jobs logs \"b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102\" --output json", - "file": "eb9f9ed47f.json" - }, - { - "match": "or jobs logs b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102 --output json", - "file": "eb9f9ed47f.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/uip-fixture.json new file mode 100644 index 0000000000..59340fc806 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LedgerExportXlsx\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-01T09:50:00.214Z\",\r\n \"EndTime\": \"2026-06-01T09:50:01.331Z\",\r\n \"CreationTime\": \"2026-06-01T09:49:59.902Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T09:50:01.288Z\",\r\n \"Message\": \"[Connect to Database] A database error occurred. The runtime resolved an empty ProviderName to the SqlClient provider (Microsoft.Data.SqlClient), but the connection string is an ACE/Excel OLE DB string (Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\\\\Finance\\\\Ledger.xlsx).\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T09:50:01.312Z\",\r\n \"Message\": \"System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\\r\\n at Microsoft.Data.SqlClient.SqlConnection..ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider)\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102\",\r\n \"StartTime\": \"2026-06-01T09:50:00.214Z\",\r\n \"EndTime\": \"2026-06-01T09:50:01.331Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"a725110e-db4d-441f-9c86-8095106f6562\",\r\n \"Info\": \"Connect to Database: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\\r\\n at Microsoft.Data.SqlClient.SqlConnection..ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider)\",\r\n \"CreationTime\": \"2026-06-01T09:49:59.902Z\",\r\n \"ReleaseName\": \"LedgerExportXlsx\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5052114,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-01T09:50:01.331Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102\",\r\n \"Id\": 671050214\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b6f7a8c9-0d1e-4f2a-9b3c-5d6e7f809102", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T09:50:01.288Z\",\r\n \"Message\": \"[Connect to Database] A database error occurred. The runtime resolved an empty ProviderName to the SqlClient provider (Microsoft.Data.SqlClient), but the connection string is an ACE/Excel OLE DB string (Provider=Microsoft.ACE.OLEDB.12.0;Data Source=C:\\\\Finance\\\\Ledger.xlsx).\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T09:50:01.312Z\",\r\n \"Message\": \"System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\\r\\n at Microsoft.Data.SqlClient.SqlConnection..ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider)\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a5e6f7b8-9c0d-4e1f-8a2b-4c5d6e7f8091\",\r\n \"Name\": \"Finance Operations\",\r\n \"Path\": \"Finance Operations\",\r\n \"Description\": \"Production folder for finance operations processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c7d8e9fa-0b1c-4d2e-8f3a-6b7c8d9e0f12\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/task.yaml index 3ec2225b6e..ed4db5eba2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-connect-database-provider-init/task.yaml @@ -28,22 +28,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/README.md new file mode 100644 index 0000000000..a4e5e5ed0a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/README.md @@ -0,0 +1,81 @@ +# Execute Non Query Failure — Driver / Client Library Not Loadable + +This scenario reproduces a runtime `Execute Non Query` failure caused by +the **database client / driver native library being absent or the wrong +bitness** on the Robot host. The SQL, the connection string, and the +schema are all correct — the provider simply cannot load its native +client, so the native load fails with `Failed to load library +(ErrorCode: 126)` / `System.DllNotFoundException` before the statement +ever reaches the database. + +## What this scenario uncovers + +**Root Cause:** The `Execute Non Query` activity targets an Oracle +database with `ProviderName = "Oracle.DataAccess.Client"`, which depends +on the Oracle Client native libraries (`OraOps12.dll` and friends). The +Robot host either lacks those libraries or has them installed at a +bitness that does not match the **process**. The project's +`targetFramework` is `Windows` (a 64-bit process), so it needs the +**64-bit** Oracle Client; a 32-bit client does not satisfy it and +produces the same `ErrorCode: 126`. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` +**Branch 4 (driver / client library not loadable)** — the same +driver/bitness family as `connect-to-database-failures.md` Branch 2. + +> **Why "Branch 4":** the `Execute Non Query` playbook has four +> branches — output-parameter size (1), unsafe SQL / parameter mapping +> (2), empty `Sql` (3), and driver/client not loadable (4). The +> `ErrorCode: 126` + `DllNotFoundException` signature is unique to +> Branch 4. The fix lives on the host, not in the workflow. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Non Query with a well-formed parameterized `UPDATE` over the `Oracle.DataAccess.Client` provider | + +The decisive evidence: + +1. The error log / job `Info`: `System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' ... Failed to load library (ErrorCode: 126)` at the `Execute Non Query` step. +2. `Main.xaml` shows `ProviderName = "Oracle.DataAccess.Client"` (a native-client provider) with a well-formed parameterized `Sql` — proving the failure is the driver, not the statement. +3. `project.json` sets `targetFramework: Windows` (a 64-bit process), anchoring the bitness requirement (64-bit client). + +## How this differs from sibling Execute Non Query branches + +| Dimension | Branch 1 (output-param size) | Branch 2 (SQL construction) | Branch 3 (empty Sql) | Branch 4 (driver load — this) | +|---|---|---|---|---| +| Error anchor | `Size property has an invalid size of 0` | `A database error occurred` + inner syntax/type error | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` / `DllNotFoundException` | +| `Sql` well-formed? | yes | no (concatenated / wrong type) | no (empty) | yes | +| Reaches the database? | yes | yes | no (ADO.NET refuses) | **no — provider never loads** | +| Where the fix lives | workflow (parameter `Size`) / package version | workflow (parameterize) | workflow (populate `Sql`) | **Robot host (install driver)** | + +This scenario is one where the failing layer is **the host +environment** — the diagnosing user has Orchestrator access only and +cannot get on the Robot machine, so the agent must reach the root cause +from evidence and hand back a **host-side install instruction** (install +the matching-bitness driver, then verify with Configure Connection) — +not a workflow or SQL change. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (Branch 4) AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) identify the missing / wrong-bitness DB driver / client library as the cause, (b) tie it to `ErrorCode 126` / `DllNotFoundException`, and (c) recommend installing the matching-bitness (64-bit) driver/client on the Robot host and verifying with Configure Connection — NOT a SQL, parameter, connection-string, or activity change + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-non-query-driver-load --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/87f83e8b25.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/87f83e8b25.json deleted file mode 100644 index 9de9137d9c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/87f83e8b25.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f", - "StartTime": "2026-05-21T08:15:02.118Z", - "EndTime": "2026-05-21T08:15:03.642Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "8a3c1e2d-4f6b-49a7-b2c8-1d5e7f3a9b4c", - "Info": "Execute Non Query: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\r\n\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)", - "CreationTime": "2026-05-21T08:15:01.804Z", - "ReleaseName": "OracleLedgerSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6041337, - "OrganizationUnitId": 7791624, - "FolderKey": "7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER2", - "OrchestratorUserIdentity": "RobotUser2", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T08:15:03.642Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f", - "Id": 671882140 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/README.md deleted file mode 100644 index 0f97cbe29e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/README.md +++ /dev/null @@ -1,80 +0,0 @@ -# Execute Non Query Failure — Driver / Client Library Not Loadable - -This scenario reproduces a runtime `Execute Non Query` failure caused by -the **database client / driver native library being absent or the wrong -bitness** on the Robot host. The SQL, the connection string, and the -schema are all correct — the provider simply cannot load its native -client, so the native load fails with `Failed to load library -(ErrorCode: 126)` / `System.DllNotFoundException` before the statement -ever reaches the database. - -## What this scenario uncovers - -**Root Cause:** The `Execute Non Query` activity targets an Oracle -database with `ProviderName = "Oracle.DataAccess.Client"`, which depends -on the Oracle Client native libraries (`OraOps12.dll` and friends). The -Robot host either lacks those libraries or has them installed at a -bitness that does not match the **process**. The project's -`targetFramework` is `Windows` (a 64-bit process), so it needs the -**64-bit** Oracle Client; a 32-bit client does not satisfy it and -produces the same `ErrorCode: 126`. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` -**Branch 4 (driver / client library not loadable)** — the same -driver/bitness family as `connect-to-database-failures.md` Branch 2. - -> **Why "Branch 4":** the `Execute Non Query` playbook has four -> branches — output-parameter size (1), unsafe SQL / parameter mapping -> (2), empty `Sql` (3), and driver/client not loadable (4). The -> `ErrorCode: 126` + `DllNotFoundException` signature is unique to -> Branch 4. The fix lives on the host, not in the workflow. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Non Query with a well-formed parameterized `UPDATE` over the `Oracle.DataAccess.Client` provider | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The error log / job `Info`: `System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' ... Failed to load library (ErrorCode: 126)` at the `Execute Non Query` step. -2. `Main.xaml` shows `ProviderName = "Oracle.DataAccess.Client"` (a native-client provider) with a well-formed parameterized `Sql` — proving the failure is the driver, not the statement. -3. `project.json` sets `targetFramework: Windows` (a 64-bit process), anchoring the bitness requirement (64-bit client). - -## How this differs from sibling Execute Non Query branches - -| Dimension | Branch 1 (output-param size) | Branch 2 (SQL construction) | Branch 3 (empty Sql) | Branch 4 (driver load — this) | -|---|---|---|---|---| -| Error anchor | `Size property has an invalid size of 0` | `A database error occurred` + inner syntax/type error | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` / `DllNotFoundException` | -| `Sql` well-formed? | yes | no (concatenated / wrong type) | no (empty) | yes | -| Reaches the database? | yes | yes | no (ADO.NET refuses) | **no — provider never loads** | -| Where the fix lives | workflow (parameter `Size`) / package version | workflow (parameterize) | workflow (populate `Sql`) | **Robot host (install driver)** | - -This scenario is one where the failing layer is **the host -environment** — the diagnosing user has Orchestrator access only and -cannot get on the Robot machine, so the agent must reach the root cause -from evidence and hand back a **host-side install instruction** (install -the matching-bitness driver, then verify with Configure Connection) — -not a workflow or SQL change. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (Branch 4) AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) identify the missing / wrong-bitness DB driver / client library as the cause, (b) tie it to `ErrorCode 126` / `DllNotFoundException`, and (c) recommend installing the matching-bitness (64-bit) driver/client on the Robot host and verifying with Configure Connection — NOT a SQL, parameter, connection-string, or activity change - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-non-query-driver-load --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/a4f4decbfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/a4f4decbfe.json deleted file mode 100644 index 26e72870b5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/a4f4decbfe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e", - "Name": "Finance Automations", - "Path": "Finance Automations", - "Description": "Production folder for finance database automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "d9a1c245-6e3b-48f7-b1c9-2a4d5e6f7a8b", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/a886f1ff6e.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/a886f1ff6e.json deleted file mode 100644 index 706313cff8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/a886f1ff6e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f", - "State": "Faulted", - "ProcessName": "OracleLedgerSync", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T08:15:02.118Z", - "EndTime": "2026-05-21T08:15:03.642Z", - "CreationTime": "2026-05-21T08:15:01.804Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/d592cdc583.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/d592cdc583.json deleted file mode 100644 index e4542da3bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/d592cdc583.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-21T08:15:03.610Z", - "Message": "[Execute Non Query] A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T08:15:03.628Z", - "Message": "Execute Non Query: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T08:15:03.640Z", - "Message": "OracleLedgerSync execution error: System.DllNotFoundException at OracleInternal.Common.ProviderConfig native client load. The Oracle Data Provider for .NET requires the Oracle Client native libraries to be installed on the host. ErrorCode 126 = the specified module could not be found." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/manifest.json deleted file mode 100644 index cc1a8dd1ef..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Non Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "a4f4decbfe.json" - }, - { - "match": "or jobs list --folder-key \"7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e\" --state Faulted --output json", - "file": "a886f1ff6e.json" - }, - { - "match": "or jobs list --folder-key 7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e --state Faulted --output json", - "file": "a886f1ff6e.json" - }, - { - "match": "or jobs get \"2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f\" --output json", - "file": "87f83e8b25.json" - }, - { - "match": "or jobs get 2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f --output json", - "file": "87f83e8b25.json" - }, - { - "match": "or jobs logs \"2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f\" --level Error --output json", - "file": "d592cdc583.json" - }, - { - "match": "or jobs logs 2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f --level Error --output json", - "file": "d592cdc583.json" - }, - { - "match": "or jobs logs \"2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f\" --output json", - "file": "d592cdc583.json" - }, - { - "match": "or jobs logs 2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f --output json", - "file": "d592cdc583.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/uip-fixture.json new file mode 100644 index 0000000000..3653658a78 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OracleLedgerSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T08:15:02.118Z\",\r\n \"EndTime\": \"2026-05-21T08:15:03.642Z\",\r\n \"CreationTime\": \"2026-05-21T08:15:01.804Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T08:15:03.610Z\",\r\n \"Message\": \"[Execute Non Query] A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T08:15:03.628Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T08:15:03.640Z\",\r\n \"Message\": \"OracleLedgerSync execution error: System.DllNotFoundException at OracleInternal.Common.ProviderConfig native client load. The Oracle Data Provider for .NET requires the Oracle Client native libraries to be installed on the host. ErrorCode 126 = the specified module could not be found.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f\",\r\n \"StartTime\": \"2026-05-21T08:15:02.118Z\",\r\n \"EndTime\": \"2026-05-21T08:15:03.642Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"8a3c1e2d-4f6b-49a7-b2c8-1d5e7f3a9b4c\",\r\n \"Info\": \"Execute Non Query: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\",\r\n \"CreationTime\": \"2026-05-21T08:15:01.804Z\",\r\n \"ReleaseName\": \"OracleLedgerSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6041337,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER2\",\r\n \"OrchestratorUserIdentity\": \"RobotUser2\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T08:15:03.642Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f\",\r\n \"Id\": 671882140\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2b8d4f6a-9c1e-4d3b-8a7f-5e2c1d3b4a6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T08:15:03.610Z\",\r\n \"Message\": \"[Execute Non Query] A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T08:15:03.628Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred. ---> System.DllNotFoundException: Unable to load DLL 'OraOps12.dll' or one of its dependencies: Failed to load library (ErrorCode: 126)\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T08:15:03.640Z\",\r\n \"Message\": \"OracleLedgerSync execution error: System.DllNotFoundException at OracleInternal.Common.ProviderConfig native client load. The Oracle Data Provider for .NET requires the Oracle Client native libraries to be installed on the host. ErrorCode 126 = the specified module could not be found.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7c3e9a1d-5b8f-4a2c-9e6d-3f1a2b4c5d6e\",\r\n \"Name\": \"Finance Automations\",\r\n \"Path\": \"Finance Automations\",\r\n \"Description\": \"Production folder for finance database automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"d9a1c245-6e3b-48f7-b1c9-2a4d5e6f7a8b\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/task.yaml index 08673056b5..6e08249e4e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-driver-load/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/README.md new file mode 100644 index 0000000000..fbb8ed0546 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/README.md @@ -0,0 +1,69 @@ +# Execute Non Query Failure — Empty Sql (CommandText not initialized) + +This scenario reproduces a runtime `Execute Non Query` failure caused by +an **empty `Sql`**. The activity's `Sql` property is bound to an +expression that resolves to `""` at runtime, so ADO.NET rejects the +command with `CommandText property has not been initialized`. The +database, connection, driver, and SQL syntax are all fine — the command +simply has no text. + +## What this scenario uncovers + +**Root Cause:** The `Execute Non Query` activity in `Main.xaml` has +`Sql="[sqlText]"`. The upstream Assign sets `sqlText` from +`dt_Config.Rows[0]["UpdateStatement"]` only when the table has rows, +falling back to `""`. At runtime `dt_Config` is empty, so `sqlText` +stays blank and the activity faults with `CommandText property has not +been initialized`. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` +(**Branch 3 — Empty `Sql`**). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — `Execute Non Query` with `Sql` bound to a variable that resolves empty | + +The decisive evidence: + +1. The `or jobs get` Info / `or jobs logs` error: `Execute Non Query: A database error occurred. CommandText property has not been initialized.` +2. The job-log trace `About to run Execute Non Query. SQL: ''` (preceded by `dt_Config row count: 0`) — proof the resolved `Sql` is empty. +3. `Main.xaml` shows `Sql="[sqlText]"` (an expression, not a literal) and the upstream Assign that leaves `sqlText` blank when the DataTable is empty. + +## How this differs from sibling Execute Non Query branches + +| Dimension | Branch 1 (output-param size) | Branch 2 (SQL syntax / params) | Branch 3 — empty Sql (this) | Branch 4 (driver load) | +|---|---|---|---|---| +| Error signature | `Size property has an invalid size of 0` | `A database error occurred` wrapping syntax/type error | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` | +| `Sql` present? | yes (proc name) | yes (malformed) | **no — resolves to ''** | yes | +| Connection opened? | yes | yes | yes | no (load fails) | +| `CommandType` | StoredProcedure | Text | **Text** | any | +| Fix layer | Parameter `Size` | Parameterize statement | **Sql expression + upstream source/scope** | Host driver/bitness | + +The agent must reject SQL-syntax error, output-parameter sizing, +driver-not-loaded, null connection, wrong activity, and database outage, +and land specifically on the empty/unset `Sql` (Branch 3). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (Branch 3) AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) identify the `Sql` as empty/unset (`CommandText property has not been initialized`), (b) cite the empty resolved SQL in the logs and the expression-bound `Sql` in source, and (c) recommend ensuring `Sql` resolves non-empty (log + fix the upstream assignment / variable scope) + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-non-query-empty-sql --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/398f066dd8.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/398f066dd8.json deleted file mode 100644 index caa21de076..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/398f066dd8.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-20T08:15:03.602Z", - "Message": "ConfigSyncProcess execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-20T08:15:04.110Z", - "Message": "Connect to Database: connection opened (Provider=Microsoft.Data.SqlClient)" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-20T08:15:04.450Z", - "Message": "[Build SQL from config row] dt_Config row count: 0" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-20T08:15:04.470Z", - "Message": "About to run Execute Non Query. SQL: ''" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-20T08:15:04.560Z", - "Message": "Execute Non Query: A database error occurred. CommandText property has not been initialized." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-20T08:15:04.580Z", - "Message": "CommandText property has not been initialized.\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/45f3c02ea1.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/45f3c02ea1.json deleted file mode 100644 index debdb53ebf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/45f3c02ea1.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d", - "Name": "Finance Automation", - "Path": "Finance Automation", - "Description": "Folder for scheduled finance back-office processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/496df6cc05.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/496df6cc05.json deleted file mode 100644 index a168e75513..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/496df6cc05.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e", - "StartTime": "2026-05-20T08:15:03.118Z", - "EndTime": "2026-05-20T08:15:04.602Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "5b6c7d8e-9f0a-4b1c-8d2e-3f4a5b6c7d8e", - "Info": "Execute Non Query: A database error occurred. CommandText property has not been initialized.\r\n\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: A database error occurred. ---> System.InvalidOperationException: CommandText property has not been initialized.", - "CreationTime": "2026-05-20T08:15:02.901Z", - "ReleaseName": "ConfigSyncProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5041337, - "OrganizationUnitId": 7791624, - "FolderKey": "7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:15:04.602Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e", - "Id": 671022914 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/README.md deleted file mode 100644 index dc5805e9bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/README.md +++ /dev/null @@ -1,68 +0,0 @@ -# Execute Non Query Failure — Empty Sql (CommandText not initialized) - -This scenario reproduces a runtime `Execute Non Query` failure caused by -an **empty `Sql`**. The activity's `Sql` property is bound to an -expression that resolves to `""` at runtime, so ADO.NET rejects the -command with `CommandText property has not been initialized`. The -database, connection, driver, and SQL syntax are all fine — the command -simply has no text. - -## What this scenario uncovers - -**Root Cause:** The `Execute Non Query` activity in `Main.xaml` has -`Sql="[sqlText]"`. The upstream Assign sets `sqlText` from -`dt_Config.Rows[0]["UpdateStatement"]` only when the table has rows, -falling back to `""`. At runtime `dt_Config` is empty, so `sqlText` -stays blank and the activity faults with `CommandText property has not -been initialized`. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` -(**Branch 3 — Empty `Sql`**). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — `Execute Non Query` with `Sql` bound to a variable that resolves empty | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The `or jobs get` Info / `or jobs logs` error: `Execute Non Query: A database error occurred. CommandText property has not been initialized.` -2. The job-log trace `About to run Execute Non Query. SQL: ''` (preceded by `dt_Config row count: 0`) — proof the resolved `Sql` is empty. -3. `Main.xaml` shows `Sql="[sqlText]"` (an expression, not a literal) and the upstream Assign that leaves `sqlText` blank when the DataTable is empty. - -## How this differs from sibling Execute Non Query branches - -| Dimension | Branch 1 (output-param size) | Branch 2 (SQL syntax / params) | Branch 3 — empty Sql (this) | Branch 4 (driver load) | -|---|---|---|---|---| -| Error signature | `Size property has an invalid size of 0` | `A database error occurred` wrapping syntax/type error | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` | -| `Sql` present? | yes (proc name) | yes (malformed) | **no — resolves to ''** | yes | -| Connection opened? | yes | yes | yes | no (load fails) | -| `CommandType` | StoredProcedure | Text | **Text** | any | -| Fix layer | Parameter `Size` | Parameterize statement | **Sql expression + upstream source/scope** | Host driver/bitness | - -The agent must reject SQL-syntax error, output-parameter sizing, -driver-not-loaded, null connection, wrong activity, and database outage, -and land specifically on the empty/unset `Sql` (Branch 3). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (Branch 3) AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) identify the `Sql` as empty/unset (`CommandText property has not been initialized`), (b) cite the empty resolved SQL in the logs and the expression-bound `Sql` in source, and (c) recommend ensuring `Sql` resolves non-empty (log + fix the upstream assignment / variable scope) - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-non-query-empty-sql --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/c938076446.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/c938076446.json deleted file mode 100644 index 74035d75f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/c938076446.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-20T08:15:04.560Z", - "Message": "Execute Non Query: A database error occurred. CommandText property has not been initialized." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-20T08:15:04.580Z", - "Message": "CommandText property has not been initialized.\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/ff61cd7b09.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/ff61cd7b09.json deleted file mode 100644 index 538d134ec8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/ff61cd7b09.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e", - "State": "Faulted", - "ProcessName": "ConfigSyncProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:15:03.118Z", - "EndTime": "2026-05-20T08:15:04.602Z", - "CreationTime": "2026-05-20T08:15:02.901Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/manifest.json deleted file mode 100644 index 1cb2bb8abe..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Non Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "45f3c02ea1.json" - }, - { - "match": "or jobs list --folder-key \"7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d\" --state Faulted --output json", - "file": "ff61cd7b09.json" - }, - { - "match": "or jobs list --folder-key 7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d --state Faulted --output json", - "file": "ff61cd7b09.json" - }, - { - "match": "or jobs get \"2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e\" --output json", - "file": "496df6cc05.json" - }, - { - "match": "or jobs get 2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e --output json", - "file": "496df6cc05.json" - }, - { - "match": "or jobs logs \"2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e\" --level Error --output json", - "file": "c938076446.json" - }, - { - "match": "or jobs logs 2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e --level Error --output json", - "file": "c938076446.json" - }, - { - "match": "or jobs logs \"2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e\" --output json", - "file": "398f066dd8.json" - }, - { - "match": "or jobs logs 2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e --output json", - "file": "398f066dd8.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/uip-fixture.json new file mode 100644 index 0000000000..7d8ce9903e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ConfigSyncProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:15:03.118Z\",\r\n \"EndTime\": \"2026-05-20T08:15:04.602Z\",\r\n \"CreationTime\": \"2026-05-20T08:15:02.901Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.560Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred. CommandText property has not been initialized.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.580Z\",\r\n \"Message\": \"CommandText property has not been initialized.\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e\",\r\n \"StartTime\": \"2026-05-20T08:15:03.118Z\",\r\n \"EndTime\": \"2026-05-20T08:15:04.602Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"5b6c7d8e-9f0a-4b1c-8d2e-3f4a5b6c7d8e\",\r\n \"Info\": \"Execute Non Query: A database error occurred. CommandText property has not been initialized.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: A database error occurred. ---> System.InvalidOperationException: CommandText property has not been initialized.\",\r\n \"CreationTime\": \"2026-05-20T08:15:02.901Z\",\r\n \"ReleaseName\": \"ConfigSyncProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5041337,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:15:04.602Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e\",\r\n \"Id\": 671022914\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2f8a4d6c-9b1e-4c3a-8d5f-6e7a8b9c0d1e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-20T08:15:03.602Z\",\r\n \"Message\": \"ConfigSyncProcess execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.110Z\",\r\n \"Message\": \"Connect to Database: connection opened (Provider=Microsoft.Data.SqlClient)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.450Z\",\r\n \"Message\": \"[Build SQL from config row] dt_Config row count: 0\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.470Z\",\r\n \"Message\": \"About to run Execute Non Query. SQL: ''\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.560Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred. CommandText property has not been initialized.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-20T08:15:04.580Z\",\r\n \"Message\": \"CommandText property has not been initialized.\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7c1e9a2b-3d4f-4a5b-8c6d-9e0f1a2b3c4d\",\r\n \"Name\": \"Finance Automation\",\r\n \"Path\": \"Finance Automation\",\r\n \"Description\": \"Folder for scheduled finance back-office processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/task.yaml index 8296c9128e..38f77e3709 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-empty-sql/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/README.md new file mode 100644 index 0000000000..93e5aab9dc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/README.md @@ -0,0 +1,72 @@ +# Execute Non Query Failure — Stored-Proc Output Parameter Size=0 (ODBC) + +This scenario reproduces a runtime `Execute Non Query` failure caused by +a **stored-procedure output parameter whose `Size` is left at `0`**, +invoked over an **ODBC** provider. The ODBC driver cannot size the value +the procedure returns, so it throws +`The Size property has an invalid size of 0` before the command runs. + +## What this scenario uncovers + +**Root Cause:** The `Execute Non Query` activity calls the stored +procedure `usp_FinalizeOrder` with `CommandType=StoredProcedure` over +`ProviderName="System.Data.Odbc"`. Its `Parameters` collection has an +`Output` parameter (`@ResultStatus`, `Direction=Output`) with +`Size="0"`. Over ODBC, output parameters need an explicit buffer size; +`Size=0` is rejected at parameter-binding time. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` +— **Branch 1** ("Stored-procedure output parameter with `Size = 0` +(ODBC)"). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Non Query stored proc over ODBC with an Output parameter `Size=0` | + +The decisive evidence: + +1. The job error (`or jobs get`): `The Size property has an invalid size of 0` with inner `System.Data.Odbc.OdbcParameter` exception. +2. The job logs show the `Execute Non Query - usp_FinalizeOrder` step running `usp_FinalizeOrder` (`CommandType=StoredProcedure`) over `System.Data.Odbc`. +3. The workflow source (`Main.xaml`) shows `CommandType=StoredProcedure`, `ProviderName="System.Data.Odbc"`, and the `@ResultStatus` parameter with `Direction=Output` and `Size="0"`. +4. `project.json` declares `UiPath.Database.Activities` `[1.3.2]`, predating the historical `1.4.0` ODBC-sizing fix. + +## How this differs from sibling Execute Non Query branches + +| Dimension | Branch 1 (this) | Branch 2 (SQL/params) | Branch 3 (empty Sql) | Branch 4 (driver load) | +|---|---|---|---|---| +| Error anchor | `The Size property has an invalid size of 0` | `A database error occurred` + syntax/type error | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` / `DllNotFoundException` | +| CommandType | StoredProcedure | Text (usually) | any | any | +| Provider | ODBC | any | any | driver missing/wrong bitness | +| Output param Size=0? | **yes** | n/a | n/a | n/a | +| Fix | set param `Size` / upgrade package | parameterize SQL / fix type | populate `Sql` | install driver at right bitness | + +The failing layer here is the **ODBC output-parameter buffer size** — +not the SQL text, not the connection, not a missing driver, not the +activity choice. The agent must recommend setting the parameter `Size` +(or upgrading `UiPath.Database.Activities`), not rewriting the SQL or +touching the connection. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the execute-non-query-failures playbook Branch 1 AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) name the `Size=0` output parameter as the cause, (b) tie it to the `StoredProcedure` CommandType over the ODBC provider, and (c) recommend setting the parameter `Size` (or upgrading `UiPath.Database.Activities`) + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-non-query-out-param-size --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/057e3b4873.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/057e3b4873.json deleted file mode 100644 index d6a470ea17..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/057e3b4873.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-21T09:14:31.305Z", - "Message": "StoredProcWriteBack execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-21T09:14:32.610Z", - "Message": "Execute Non Query - usp_FinalizeOrder executing stored procedure 'usp_FinalizeOrder' (CommandType=StoredProcedure) over provider 'System.Data.Odbc'" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:32.840Z", - "Message": "[Execute Non Query - usp_FinalizeOrder] A database error occurred. The Size property has an invalid size of 0." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:32.855Z", - "Message": "A database error occurred. The Size property has an invalid size of 0.\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query - usp_FinalizeOrder\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/2a1470785d.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/2a1470785d.json deleted file mode 100644 index e2ee64dbc1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/2a1470785d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d", - "StartTime": "2026-05-21T09:14:31.220Z", - "EndTime": "2026-05-21T09:14:32.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "8d1c2f3a-4b5e-4c6d-9a7b-2e3f4a5b6c7d", - "Info": "Execute Non Query - usp_FinalizeOrder: A database error occurred. The Size property has an invalid size of 0.\r\n\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query - usp_FinalizeOrder\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: The Size property has an invalid size of 0.\r\n at System.Data.Odbc.OdbcParameter.GetParameterSize(...)\r\n at System.Data.Odbc.OdbcCommand.ExecuteNonQuery()", - "CreationTime": "2026-05-21T09:14:30.991Z", - "ReleaseName": "StoredProcWriteBack", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5048113, - "OrganizationUnitId": 7791624, - "FolderKey": "7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T09:14:32.870Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d", - "Id": 671103847 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/568af2d743.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/568af2d743.json deleted file mode 100644 index 5544f1244f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/568af2d743.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e", - "Name": "Finance Automations", - "Path": "Finance Automations", - "Description": "Production folder for finance back-office processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c5a1d8f3-4b7e-4a2c-9f6d-1e3b2a4c5d6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/648d46929d.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/648d46929d.json deleted file mode 100644 index ce5dd5a277..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/648d46929d.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:32.840Z", - "Message": "[Execute Non Query - usp_FinalizeOrder] A database error occurred. The Size property has an invalid size of 0." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:32.855Z", - "Message": "A database error occurred. The Size property has an invalid size of 0.\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query - usp_FinalizeOrder\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/README.md deleted file mode 100644 index 44bafa72b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/README.md +++ /dev/null @@ -1,71 +0,0 @@ -# Execute Non Query Failure — Stored-Proc Output Parameter Size=0 (ODBC) - -This scenario reproduces a runtime `Execute Non Query` failure caused by -a **stored-procedure output parameter whose `Size` is left at `0`**, -invoked over an **ODBC** provider. The ODBC driver cannot size the value -the procedure returns, so it throws -`The Size property has an invalid size of 0` before the command runs. - -## What this scenario uncovers - -**Root Cause:** The `Execute Non Query` activity calls the stored -procedure `usp_FinalizeOrder` with `CommandType=StoredProcedure` over -`ProviderName="System.Data.Odbc"`. Its `Parameters` collection has an -`Output` parameter (`@ResultStatus`, `Direction=Output`) with -`Size="0"`. Over ODBC, output parameters need an explicit buffer size; -`Size=0` is rejected at parameter-binding time. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` -— **Branch 1** ("Stored-procedure output parameter with `Size = 0` -(ODBC)"). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Non Query stored proc over ODBC with an Output parameter `Size=0` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented Branch 1 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The job error (`or jobs get`): `The Size property has an invalid size of 0` with inner `System.Data.Odbc.OdbcParameter` exception. -2. The job logs show the `Execute Non Query - usp_FinalizeOrder` step running `usp_FinalizeOrder` (`CommandType=StoredProcedure`) over `System.Data.Odbc`. -3. The workflow source (`Main.xaml`) shows `CommandType=StoredProcedure`, `ProviderName="System.Data.Odbc"`, and the `@ResultStatus` parameter with `Direction=Output` and `Size="0"`. -4. `project.json` declares `UiPath.Database.Activities` `[1.3.2]`, predating the historical `1.4.0` ODBC-sizing fix. - -## How this differs from sibling Execute Non Query branches - -| Dimension | Branch 1 (this) | Branch 2 (SQL/params) | Branch 3 (empty Sql) | Branch 4 (driver load) | -|---|---|---|---|---| -| Error anchor | `The Size property has an invalid size of 0` | `A database error occurred` + syntax/type error | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` / `DllNotFoundException` | -| CommandType | StoredProcedure | Text (usually) | any | any | -| Provider | ODBC | any | any | driver missing/wrong bitness | -| Output param Size=0? | **yes** | n/a | n/a | n/a | -| Fix | set param `Size` / upgrade package | parameterize SQL / fix type | populate `Sql` | install driver at right bitness | - -The failing layer here is the **ODBC output-parameter buffer size** — -not the SQL text, not the connection, not a missing driver, not the -activity choice. The agent must recommend setting the parameter `Size` -(or upgrading `UiPath.Database.Activities`), not rewriting the SQL or -touching the connection. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the execute-non-query-failures playbook Branch 1 AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) name the `Size=0` output parameter as the cause, (b) tie it to the `StoredProcedure` CommandType over the ODBC provider, and (c) recommend setting the parameter `Size` (or upgrading `UiPath.Database.Activities`) - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-non-query-out-param-size --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/f059f7355d.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/f059f7355d.json deleted file mode 100644 index c02fe7f8de..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/f059f7355d.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d", - "State": "Faulted", - "ProcessName": "StoredProcWriteBack", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:14:31.220Z", - "EndTime": "2026-05-21T09:14:32.870Z", - "CreationTime": "2026-05-21T09:14:30.991Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/manifest.json deleted file mode 100644 index 74adbf217c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Non Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "568af2d743.json" - }, - { - "match": "or jobs list --folder-key \"7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e\" --state Faulted --output json", - "file": "f059f7355d.json" - }, - { - "match": "or jobs list --folder-key 7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e --state Faulted --output json", - "file": "f059f7355d.json" - }, - { - "match": "or jobs get \"3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d\" --output json", - "file": "2a1470785d.json" - }, - { - "match": "or jobs get 3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d --output json", - "file": "2a1470785d.json" - }, - { - "match": "or jobs logs \"3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d\" --level Error --output json", - "file": "648d46929d.json" - }, - { - "match": "or jobs logs 3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d --level Error --output json", - "file": "648d46929d.json" - }, - { - "match": "or jobs logs \"3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d\" --output json", - "file": "057e3b4873.json" - }, - { - "match": "or jobs logs 3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d --output json", - "file": "057e3b4873.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/uip-fixture.json new file mode 100644 index 0000000000..2bc5981ead --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"StoredProcWriteBack\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:14:31.220Z\",\r\n \"EndTime\": \"2026-05-21T09:14:32.870Z\",\r\n \"CreationTime\": \"2026-05-21T09:14:30.991Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:32.840Z\",\r\n \"Message\": \"[Execute Non Query - usp_FinalizeOrder] A database error occurred. The Size property has an invalid size of 0.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:32.855Z\",\r\n \"Message\": \"A database error occurred. The Size property has an invalid size of 0.\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query - usp_FinalizeOrder\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d\",\r\n \"StartTime\": \"2026-05-21T09:14:31.220Z\",\r\n \"EndTime\": \"2026-05-21T09:14:32.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"8d1c2f3a-4b5e-4c6d-9a7b-2e3f4a5b6c7d\",\r\n \"Info\": \"Execute Non Query - usp_FinalizeOrder: A database error occurred. The Size property has an invalid size of 0.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query - usp_FinalizeOrder\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: The Size property has an invalid size of 0.\\r\\n at System.Data.Odbc.OdbcParameter.GetParameterSize(...)\\r\\n at System.Data.Odbc.OdbcCommand.ExecuteNonQuery()\",\r\n \"CreationTime\": \"2026-05-21T09:14:30.991Z\",\r\n \"ReleaseName\": \"StoredProcWriteBack\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5048113,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T09:14:32.870Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d\",\r\n \"Id\": 671103847\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3f8a1c6d-2b4e-4a9c-9d7b-5e1f2a3b4c6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-21T09:14:31.305Z\",\r\n \"Message\": \"StoredProcWriteBack execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:32.610Z\",\r\n \"Message\": \"Execute Non Query - usp_FinalizeOrder executing stored procedure 'usp_FinalizeOrder' (CommandType=StoredProcedure) over provider 'System.Data.Odbc'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:32.840Z\",\r\n \"Message\": \"[Execute Non Query - usp_FinalizeOrder] A database error occurred. The Size property has an invalid size of 0.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:32.855Z\",\r\n \"Message\": \"A database error occurred. The Size property has an invalid size of 0.\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query - usp_FinalizeOrder\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7b2e9c4d-3a6f-4d1b-8e5c-2f9a1b3c4d5e\",\r\n \"Name\": \"Finance Automations\",\r\n \"Path\": \"Finance Automations\",\r\n \"Description\": \"Production folder for finance back-office processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c5a1d8f3-4b7e-4a2c-9f6d-1e3b2a4c5d6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/task.yaml index d3ab985aa8..170dbdb717 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-out-param-size/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/README.md new file mode 100644 index 0000000000..2f6db88d1d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/README.md @@ -0,0 +1,92 @@ +# Execute Non Query Failure — Unsafe SQL Construction (Branch 2) + +This scenario reproduces a runtime `Execute Non Query` failure caused by +**building SQL with string concatenation**. The activity concatenates +UiPath variables straight into an `INSERT`; at runtime one variable holds +a value with an apostrophe (`O'Brien`), the apostrophe closes the SQL +string literal early, and the provider rejects the malformed statement. +Orchestrator surfaces `Execute Non Query: A database error occurred` +wrapping a `Microsoft.Data.SqlClient` `SqlException`: +`Unclosed quotation mark after the character string ')'.` / +`Incorrect syntax near 'Brien'.` + +## What this scenario uncovers + +**Root Cause:** The `Execute Non Query` activity's `Sql` is built as +`"INSERT INTO Customers (Name, City) VALUES ('" + in_CustomerName + "', '" + in_City + "')"`. +With `in_CustomerName = "O'Brien"` the resolved SQL becomes +`INSERT INTO Customers (Name, City) VALUES ('O'Brien', 'Austin')` — the +embedded apostrophe breaks the literal, so SQL Server fails to parse it. +The fix is to **parameterize**: use a static +`INSERT INTO Customers (Name, City) VALUES (@name, @city)` and map +`in_CustomerName` → `@name`, `in_City` → `@city` via the activity's +Parameters collection (the provider handles quoting/typing). This also +closes the SQL-injection risk. Escaping/doubling the apostrophe is **not** +the correct fix. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` +(BRANCH 2 — unsafe SQL construction / parameter mapping). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Non Query whose `Sql` is a concatenation expression, behind a valid Connect/Assign structure, with no Parameters collection | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Sales Operations` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `CustomerUpsert` job. +3. `uip or jobs get --output json` → `Info` carries `Execute Non Query: A database error occurred` wrapping `Unclosed quotation mark after the character string ')'. Incorrect syntax near 'Brien'.` +4. `uip or jobs logs --output json` → a Trace line showing the **resolved** SQL `... VALUES ('O'Brien', 'Austin')` — the smoking gun proving the apostrophe broke the literal. +5. `process/Main.xaml` → the `Execute Non Query` `Sql` is the concatenation expression `"...VALUES ('" + in_CustomerName + "', '" + in_City + "')"` with `in_CustomerName` assigned `O'Brien`, and no Parameters collection. + +## How this differs from sibling branches + +| Dimension | branch 1 (output param Size=0) | branch 2 (concatenation) (this) | branch 3 (empty Sql) | branch 4 (driver load) | +|---|---|---|---|---| +| Connection opens? | yes | **yes** | yes | n/a | +| Error signature | `The Size property has an invalid size of 0` | **`A database error occurred` + `Unclosed quotation mark` / `Incorrect syntax near ...`** | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` / `DllNotFoundException` | +| Where the fault is | output-parameter sizing (ODBC) | **the SQL text** | blank/`""` Sql expression | host driver/client + bitness | +| Fix | set output param `Size` | **parameterize via Parameters collection** | populate the Sql / guard empty | install the driver at matching bitness | + +The failing layer here is the **statement text**, not the connection, a +driver, an empty command, or an output-parameter size. The agent must +recommend parameterizing with named `@parameters` via the Parameters +collection — not escaping/doubling the apostrophe, not switching to +`Execute Query`, not blaming the connection or the database. + +## Synthetic note + +the documented branch-2 signature in `execute-non-query-failures.md`, not +captured from a real tenant. All identifiers (folder key, job key, +hostname, account) are synthetic. Replace with verbatim captures from a +real `.local/investigations/` before treating this as a regression signal. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-non-query-failures playbook (branch 2) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) name the unclosed-quote / syntax error from string concatenation, (b) attribute it to building the Sql by concatenating an apostrophe-bearing value (`O'Brien`) and note the injection risk, and (c) recommend parameterizing with named `@parameters` via the Parameters collection. + +The `llm_judge` grades only the agent's final response and tool-call +summary against `RESOLUTION.md`. It does **not** inspect internal +investigation state (`.local/investigations/`, `state.json`, +`hypotheses.json`). + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-non-query-unsafe-sql --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/00b8ff12ee.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/00b8ff12ee.json deleted file mode 100644 index c9c13740f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/00b8ff12ee.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304", - "State": "Faulted", - "ProcessName": "CustomerUpsert", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-01T11:20:00.118Z", - "EndTime": "2026-06-01T11:20:01.430Z", - "CreationTime": "2026-06-01T11:19:59.905Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/2b486354cf.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/2b486354cf.json deleted file mode 100644 index d5b6440a65..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/2b486354cf.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304", - "StartTime": "2026-06-01T11:20:00.118Z", - "EndTime": "2026-06-01T11:20:01.430Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7c1d2e3f-4a5b-4c6d-9e8f-1a2b3c4d5e6f", - "Info": "Execute Non Query: A database error occurred.\r\n\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Execute Non Query: A database error occurred. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Unclosed quotation mark after the character string ')'.\r\nIncorrect syntax near 'Brien'.\r\n at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()\r\n at UiPath.Database.DatabaseConnection.ExecuteNonQuery(String sql, Dictionary`2 parameters, Int32 timeout)\r\n at UiPath.Database.Activities.ExecuteNonQuery.ExecuteAsync()\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-06-01T11:19:59.905Z", - "ReleaseName": "CustomerUpsert", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5041992, - "OrganizationUnitId": 7791624, - "FolderKey": "c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-01T11:20:01.430Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304", - "Id": 671204418 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/3656c3eacf.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/3656c3eacf.json deleted file mode 100644 index c516c2cb8a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/3656c3eacf.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-01T11:20:01.389Z", - "Message": "Execute Non Query: A database error occurred.\r\nMicrosoft.Data.SqlClient.SqlException: Unclosed quotation mark after the character string ')'.\r\nIncorrect syntax near 'Brien'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-01T11:20:01.406Z", - "Message": "Execute Non Query: A database error occurred.\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/8082ea9290.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/8082ea9290.json deleted file mode 100644 index 5067fa35f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/8082ea9290.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203", - "Name": "Sales Operations", - "Path": "Sales Operations", - "Description": "Folder hosting the customer upsert and CRM sync processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/README.md deleted file mode 100644 index 1b771e7661..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/README.md +++ /dev/null @@ -1,92 +0,0 @@ -# Execute Non Query Failure — Unsafe SQL Construction (Branch 2) - -This scenario reproduces a runtime `Execute Non Query` failure caused by -**building SQL with string concatenation**. The activity concatenates -UiPath variables straight into an `INSERT`; at runtime one variable holds -a value with an apostrophe (`O'Brien`), the apostrophe closes the SQL -string literal early, and the provider rejects the malformed statement. -Orchestrator surfaces `Execute Non Query: A database error occurred` -wrapping a `Microsoft.Data.SqlClient` `SqlException`: -`Unclosed quotation mark after the character string ')'.` / -`Incorrect syntax near 'Brien'.` - -## What this scenario uncovers - -**Root Cause:** The `Execute Non Query` activity's `Sql` is built as -`"INSERT INTO Customers (Name, City) VALUES ('" + in_CustomerName + "', '" + in_City + "')"`. -With `in_CustomerName = "O'Brien"` the resolved SQL becomes -`INSERT INTO Customers (Name, City) VALUES ('O'Brien', 'Austin')` — the -embedded apostrophe breaks the literal, so SQL Server fails to parse it. -The fix is to **parameterize**: use a static -`INSERT INTO Customers (Name, City) VALUES (@name, @city)` and map -`in_CustomerName` → `@name`, `in_City` → `@city` via the activity's -Parameters collection (the provider handles quoting/typing). This also -closes the SQL-injection risk. Escaping/doubling the apostrophe is **not** -the correct fix. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-non-query-failures.md` -(BRANCH 2 — unsafe SQL construction / parameter mapping). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Non Query whose `Sql` is a concatenation expression, behind a valid Connect/Assign structure, with no Parameters collection | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Sales Operations` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `CustomerUpsert` job. -3. `uip or jobs get --output json` → `Info` carries `Execute Non Query: A database error occurred` wrapping `Unclosed quotation mark after the character string ')'. Incorrect syntax near 'Brien'.` -4. `uip or jobs logs --output json` → a Trace line showing the **resolved** SQL `... VALUES ('O'Brien', 'Austin')` — the smoking gun proving the apostrophe broke the literal. -5. `process/Main.xaml` → the `Execute Non Query` `Sql` is the concatenation expression `"...VALUES ('" + in_CustomerName + "', '" + in_City + "')"` with `in_CustomerName` assigned `O'Brien`, and no Parameters collection. - -## How this differs from sibling branches - -| Dimension | branch 1 (output param Size=0) | branch 2 (concatenation) (this) | branch 3 (empty Sql) | branch 4 (driver load) | -|---|---|---|---|---| -| Connection opens? | yes | **yes** | yes | n/a | -| Error signature | `The Size property has an invalid size of 0` | **`A database error occurred` + `Unclosed quotation mark` / `Incorrect syntax near ...`** | `CommandText property has not been initialized` | `Failed to load library (ErrorCode: 126)` / `DllNotFoundException` | -| Where the fault is | output-parameter sizing (ODBC) | **the SQL text** | blank/`""` Sql expression | host driver/client + bitness | -| Fix | set output param `Size` | **parameterize via Parameters collection** | populate the Sql / guard empty | install the driver at matching bitness | - -The failing layer here is the **statement text**, not the connection, a -driver, an empty command, or an output-parameter size. The agent must -recommend parameterizing with named `@parameters` via the Parameters -collection — not escaping/doubling the apostrophe, not switching to -`Execute Query`, not blaming the connection or the database. - -## Synthetic note - -The fixtures under `fixtures/mocks/responses/` are **hand-authored** from -the documented branch-2 signature in `execute-non-query-failures.md`, not -captured from a real tenant. All identifiers (folder key, job key, -hostname, account) are synthetic. Replace with verbatim captures from a -real `.local/investigations/` before treating this as a regression signal. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-non-query-failures playbook (branch 2) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) name the unclosed-quote / syntax error from string concatenation, (b) attribute it to building the Sql by concatenating an apostrophe-bearing value (`O'Brien`) and note the injection risk, and (c) recommend parameterizing with named `@parameters` via the Parameters collection. - -The `llm_judge` grades only the agent's final response and tool-call -summary against `RESOLUTION.md`. It does **not** inspect internal -investigation state (`.local/investigations/`, `state.json`, -`hypotheses.json`). - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-non-query-unsafe-sql --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/c3079b71a3.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/c3079b71a3.json deleted file mode 100644 index 306f6d2156..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/c3079b71a3.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-01T11:20:00.940Z", - "Message": "CustomerUpsert execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-01T11:20:01.290Z", - "Message": "Execute Non Query: executing INSERT INTO Customers (Name, City) VALUES ('O'Brien', 'Austin') against connection 'Server=sqlsales01;Database=CRM'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-01T11:20:01.389Z", - "Message": "Execute Non Query: A database error occurred.\r\nMicrosoft.Data.SqlClient.SqlException: Unclosed quotation mark after the character string ')'.\r\nIncorrect syntax near 'Brien'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-01T11:20:01.406Z", - "Message": "Execute Non Query: A database error occurred.\r\nin Main.xaml\r\n at ExecuteNonQuery \"Execute Non Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/manifest.json deleted file mode 100644 index 876a425a06..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Non Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "8082ea9290.json" - }, - { - "match": "or jobs list --folder-key \"c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203\" --state Faulted --output json", - "file": "00b8ff12ee.json" - }, - { - "match": "or jobs list --folder-key c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203 --state Faulted --output json", - "file": "00b8ff12ee.json" - }, - { - "match": "or jobs get \"d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304\" --output json", - "file": "2b486354cf.json" - }, - { - "match": "or jobs get d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304 --output json", - "file": "2b486354cf.json" - }, - { - "match": "or jobs logs \"d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304\" --level Error --output json", - "file": "3656c3eacf.json" - }, - { - "match": "or jobs logs d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304 --level Error --output json", - "file": "3656c3eacf.json" - }, - { - "match": "or jobs logs \"d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304\" --output json", - "file": "c3079b71a3.json" - }, - { - "match": "or jobs logs d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304 --output json", - "file": "c3079b71a3.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/uip-fixture.json new file mode 100644 index 0000000000..a9b1c77f28 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerUpsert\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-01T11:20:00.118Z\",\r\n \"EndTime\": \"2026-06-01T11:20:01.430Z\",\r\n \"CreationTime\": \"2026-06-01T11:19:59.905Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T11:20:01.389Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred.\\r\\nMicrosoft.Data.SqlClient.SqlException: Unclosed quotation mark after the character string ')'.\\r\\nIncorrect syntax near 'Brien'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T11:20:01.406Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred.\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304\",\r\n \"StartTime\": \"2026-06-01T11:20:00.118Z\",\r\n \"EndTime\": \"2026-06-01T11:20:01.430Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7c1d2e3f-4a5b-4c6d-9e8f-1a2b3c4d5e6f\",\r\n \"Info\": \"Execute Non Query: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Execute Non Query: A database error occurred. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Unclosed quotation mark after the character string ')'.\\r\\nIncorrect syntax near 'Brien'.\\r\\n at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()\\r\\n at UiPath.Database.DatabaseConnection.ExecuteNonQuery(String sql, Dictionary`2 parameters, Int32 timeout)\\r\\n at UiPath.Database.Activities.ExecuteNonQuery.ExecuteAsync()\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-06-01T11:19:59.905Z\",\r\n \"ReleaseName\": \"CustomerUpsert\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5041992,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-01T11:20:01.430Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304\",\r\n \"Id\": 671204418\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d8b9c0e1-2f3a-4b4c-9d5e-7f8091020304", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-01T11:20:00.940Z\",\r\n \"Message\": \"CustomerUpsert execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-01T11:20:01.290Z\",\r\n \"Message\": \"Execute Non Query: executing INSERT INTO Customers (Name, City) VALUES ('O'Brien', 'Austin') against connection 'Server=sqlsales01;Database=CRM'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T11:20:01.389Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred.\\r\\nMicrosoft.Data.SqlClient.SqlException: Unclosed quotation mark after the character string ')'.\\r\\nIncorrect syntax near 'Brien'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-01T11:20:01.406Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred.\\r\\nin Main.xaml\\r\\n at ExecuteNonQuery \\\"Execute Non Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c7a8b9d0-1e2f-4a3b-8c4d-6e7f80910203\",\r\n \"Name\": \"Sales Operations\",\r\n \"Path\": \"Sales Operations\",\r\n \"Description\": \"Folder hosting the customer upsert and CRM sync processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/task.yaml index 39c35cb2e2..6545015f61 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-non-query-unsafe-sql/task.yaml @@ -31,22 +31,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/README.md new file mode 100644 index 0000000000..1a09b9aaec --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/README.md @@ -0,0 +1,77 @@ +# Execute Query Failure — CLR-Level Crash (`0xE0434352`) from an Oversized Result Set + +This scenario reproduces a runtime `Execute Query` failure that crashes +the **managed runtime** rather than raising a clean activity exception. +The activity runs an unbounded `SELECT * FROM Transactions` over a +multi-million-row table and materializes the whole result set into a +single `DataTable`, exhausting the Robot process memory. The job +terminates at the **process level** with exit code `0xE0434352` and **no +clean activity-level exception** in the workflow log. + +## What this scenario uncovers + +**Root Cause:** The `Execute Query` activity's `Sql` is the unbounded +literal `"SELECT * FROM Transactions"` — no `TOP`/`FETCH FIRST`/`LIMIT`, +no `WHERE` — over a huge table. Loading every row into one `DataTable` +exhausts memory; the CLR crashes and the job exits with `0xE0434352`. +`0xE0434352` is the **generic .NET unhandled-exception SEH code**, not a +DB-specific code on its own; the database context (the oversized SELECT) +narrows it. The fix is to **bound the result set** in SQL +(`TOP`/`FETCH FIRST`/`LIMIT`) and **page** through large data; update +`UiPath.Database.Activities` if a provider-path bug is implicated. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 6 — CLR-level crash `0xE0434352`). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Query whose `Sql` is an unbounded `SELECT *`, behind a valid Connect/Execute structure | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Finance Automation` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `TransactionExport` job. +3. `uip or jobs get --output json` → `Info` reports a process-level termination with exit code `0xE0434352` and **no activity-level exception** — the distinguishing Branch 6 shape. +4. `uip or jobs logs --output json` → the connection opened, `Execute Query` started against `SELECT * FROM Transactions`, a Trace line notes it was "still reading result set into DataTable (no row limit set)", then the process died. +5. `process/Main.xaml` → the `Execute Query` `Sql` is the unbounded `"SELECT * FROM Transactions"` over a multi-million-row table. + +## How this differs from sibling branches + +| Dimension | branch 1 (null connection) | branch 3 (concatenation) | branch 5 (timeout) | branch 6 (CLR crash) (this) | +|---|---|---|---|---| +| Connection opens? | no | yes | yes | **yes** | +| Error signature | `NullReferenceException` | `A database error occurred` + `Incorrect syntax near ...` | `Timeout expired` | **exit `0xE0434352`, no clean activity exception** | +| Where the fault is | wiring/scope | the SQL text | DB-side duration | **process memory (data volume)** | +| Fix | wire/scope the connection | parameterize the query | raise `TimeoutMS` / fix the query | **bound the result set (`TOP`/`FETCH`/`LIMIT`) + page** | + +The failing layer here is the **Robot process memory** driven by the +**data volume**, not the connection, the SQL syntax, the driver, or a +timeout. The crash is process-level (`0xE0434352`) with no clean activity +exception — the distinguishing evidence shape. The agent must recommend +bounding the result set in SQL and paging — not raising a timeout, not +parameterizing, not blaming the database. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (branch 6) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) recognize the unbounded `SELECT *` over a huge table materializing into a `DataTable`, (b) attribute the failure to memory exhaustion surfacing as the `0xE0434352` CLR / process-level crash with no clean activity exception, and (c) recommend bounding the result set with `TOP`/`FETCH FIRST`/`LIMIT` and paging (and updating `UiPath.Database.Activities` where a provider-path bug is implicated). + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-query-clr-crash --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/05ac9f01ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/05ac9f01ab.json deleted file mode 100644 index 4c94c1cbfa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/05ac9f01ab.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b", - "StartTime": "2026-05-22T02:05:11.402Z", - "EndTime": "2026-05-22T02:06:48.913Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9a3c1e2f-5b6d-4e7a-8c9f-2d3e4f5a6b7c", - "Info": "The job terminated unexpectedly. The robot execution process exited with code 0xE0434352. No activity-level exception was recorded in the workflow log before the process ended.\r\n\r\nReleaseName: TransactionExport\r\nLast logged activity before termination: Execute Query \"Execute Query\" (Main.xaml).", - "CreationTime": "2026-05-22T02:05:10.870Z", - "ReleaseName": "TransactionExport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5042118, - "OrganizationUnitId": 7791624, - "FolderKey": "5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER2", - "OrchestratorUserIdentity": "RobotUser2", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-22T02:06:48.913Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b", - "Id": 671209902 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/16bf822011.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/16bf822011.json deleted file mode 100644 index 6ddbdf617c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/16bf822011.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-22T02:05:12.180Z", - "Message": "TransactionExport execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-22T02:05:12.940Z", - "Message": "Connect to Database: opened connection 'Server=sqlprod02;Database=Finance' using provider Microsoft.Data.SqlClient" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-22T02:05:13.220Z", - "Message": "Execute Query: executing SQL against connection 'Server=sqlprod02;Database=Finance': SELECT * FROM Transactions" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-22T02:05:38.502Z", - "Message": "Execute Query: still reading result set into DataTable (no row limit set)..." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-22T02:06:48.871Z", - "Message": "The robot execution process terminated unexpectedly with exit code 0xE0434352. No activity-level exception was caught; the process ended while Execute Query \"Execute Query\" was materializing its result set." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/3082b0f837.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/3082b0f837.json deleted file mode 100644 index 25584dd5d3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/3082b0f837.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d", - "Name": "Finance Automation", - "Path": "Finance Automation", - "Description": "Folder hosting the finance export and reconciliation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/46e0925fd5.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/46e0925fd5.json deleted file mode 100644 index f6360cb07f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/46e0925fd5.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-22T02:06:48.871Z", - "Message": "The robot execution process terminated unexpectedly with exit code 0xE0434352. No activity-level exception was caught; the process ended while Execute Query \"Execute Query\" was materializing its result set." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/701dfa3277.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/701dfa3277.json deleted file mode 100644 index a387363bfa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/701dfa3277.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b", - "State": "Faulted", - "ProcessName": "TransactionExport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T02:05:11.402Z", - "EndTime": "2026-05-22T02:06:48.913Z", - "CreationTime": "2026-05-22T02:05:10.870Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/README.md deleted file mode 100644 index 0e0a1f541c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/README.md +++ /dev/null @@ -1,76 +0,0 @@ -# Execute Query Failure — CLR-Level Crash (`0xE0434352`) from an Oversized Result Set - -This scenario reproduces a runtime `Execute Query` failure that crashes -the **managed runtime** rather than raising a clean activity exception. -The activity runs an unbounded `SELECT * FROM Transactions` over a -multi-million-row table and materializes the whole result set into a -single `DataTable`, exhausting the Robot process memory. The job -terminates at the **process level** with exit code `0xE0434352` and **no -clean activity-level exception** in the workflow log. - -## What this scenario uncovers - -**Root Cause:** The `Execute Query` activity's `Sql` is the unbounded -literal `"SELECT * FROM Transactions"` — no `TOP`/`FETCH FIRST`/`LIMIT`, -no `WHERE` — over a huge table. Loading every row into one `DataTable` -exhausts memory; the CLR crashes and the job exits with `0xE0434352`. -`0xE0434352` is the **generic .NET unhandled-exception SEH code**, not a -DB-specific code on its own; the database context (the oversized SELECT) -narrows it. The fix is to **bound the result set** in SQL -(`TOP`/`FETCH FIRST`/`LIMIT`) and **page** through large data; update -`UiPath.Database.Activities` if a provider-path bug is implicated. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 6 — CLR-level crash `0xE0434352`). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Query whose `Sql` is an unbounded `SELECT *`, behind a valid Connect/Execute structure | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-6 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Finance Automation` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `TransactionExport` job. -3. `uip or jobs get --output json` → `Info` reports a process-level termination with exit code `0xE0434352` and **no activity-level exception** — the distinguishing Branch 6 shape. -4. `uip or jobs logs --output json` → the connection opened, `Execute Query` started against `SELECT * FROM Transactions`, a Trace line notes it was "still reading result set into DataTable (no row limit set)", then the process died. -5. `process/Main.xaml` → the `Execute Query` `Sql` is the unbounded `"SELECT * FROM Transactions"` over a multi-million-row table. - -## How this differs from sibling branches - -| Dimension | branch 1 (null connection) | branch 3 (concatenation) | branch 5 (timeout) | branch 6 (CLR crash) (this) | -|---|---|---|---|---| -| Connection opens? | no | yes | yes | **yes** | -| Error signature | `NullReferenceException` | `A database error occurred` + `Incorrect syntax near ...` | `Timeout expired` | **exit `0xE0434352`, no clean activity exception** | -| Where the fault is | wiring/scope | the SQL text | DB-side duration | **process memory (data volume)** | -| Fix | wire/scope the connection | parameterize the query | raise `TimeoutMS` / fix the query | **bound the result set (`TOP`/`FETCH`/`LIMIT`) + page** | - -The failing layer here is the **Robot process memory** driven by the -**data volume**, not the connection, the SQL syntax, the driver, or a -timeout. The crash is process-level (`0xE0434352`) with no clean activity -exception — the distinguishing evidence shape. The agent must recommend -bounding the result set in SQL and paging — not raising a timeout, not -parameterizing, not blaming the database. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (branch 6) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) recognize the unbounded `SELECT *` over a huge table materializing into a `DataTable`, (b) attribute the failure to memory exhaustion surfacing as the `0xE0434352` CLR / process-level crash with no clean activity exception, and (c) recommend bounding the result set with `TOP`/`FETCH FIRST`/`LIMIT` and paging (and updating `UiPath.Database.Activities` where a provider-path bug is implicated). - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-query-clr-crash --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/manifest.json deleted file mode 100644 index ebd6dbddaa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "3082b0f837.json" - }, - { - "match": "or jobs list --folder-key \"5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d\" --state Faulted --output json", - "file": "701dfa3277.json" - }, - { - "match": "or jobs list --folder-key 5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d --state Faulted --output json", - "file": "701dfa3277.json" - }, - { - "match": "or jobs get \"2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b\" --output json", - "file": "05ac9f01ab.json" - }, - { - "match": "or jobs get 2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b --output json", - "file": "05ac9f01ab.json" - }, - { - "match": "or jobs logs \"2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b\" --level Error --output json", - "file": "46e0925fd5.json" - }, - { - "match": "or jobs logs 2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b --level Error --output json", - "file": "46e0925fd5.json" - }, - { - "match": "or jobs logs \"2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b\" --output json", - "file": "16bf822011.json" - }, - { - "match": "or jobs logs 2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b --output json", - "file": "16bf822011.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/uip-fixture.json new file mode 100644 index 0000000000..109a8a6f0b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TransactionExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T02:05:11.402Z\",\r\n \"EndTime\": \"2026-05-22T02:06:48.913Z\",\r\n \"CreationTime\": \"2026-05-22T02:05:10.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T02:06:48.871Z\",\r\n \"Message\": \"The robot execution process terminated unexpectedly with exit code 0xE0434352. No activity-level exception was caught; the process ended while Execute Query \\\"Execute Query\\\" was materializing its result set.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b\",\r\n \"StartTime\": \"2026-05-22T02:05:11.402Z\",\r\n \"EndTime\": \"2026-05-22T02:06:48.913Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9a3c1e2f-5b6d-4e7a-8c9f-2d3e4f5a6b7c\",\r\n \"Info\": \"The job terminated unexpectedly. The robot execution process exited with code 0xE0434352. No activity-level exception was recorded in the workflow log before the process ended.\\r\\n\\r\\nReleaseName: TransactionExport\\r\\nLast logged activity before termination: Execute Query \\\"Execute Query\\\" (Main.xaml).\",\r\n \"CreationTime\": \"2026-05-22T02:05:10.870Z\",\r\n \"ReleaseName\": \"TransactionExport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5042118,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER2\",\r\n \"OrchestratorUserIdentity\": \"RobotUser2\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-22T02:06:48.913Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b\",\r\n \"Id\": 671209902\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2f6a8c4b-1d9e-4f3a-8b7c-5e6d2a1f9c3b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-22T02:05:12.180Z\",\r\n \"Message\": \"TransactionExport execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-22T02:05:12.940Z\",\r\n \"Message\": \"Connect to Database: opened connection 'Server=sqlprod02;Database=Finance' using provider Microsoft.Data.SqlClient\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-22T02:05:13.220Z\",\r\n \"Message\": \"Execute Query: executing SQL against connection 'Server=sqlprod02;Database=Finance': SELECT * FROM Transactions\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-22T02:05:38.502Z\",\r\n \"Message\": \"Execute Query: still reading result set into DataTable (no row limit set)...\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T02:06:48.871Z\",\r\n \"Message\": \"The robot execution process terminated unexpectedly with exit code 0xE0434352. No activity-level exception was caught; the process ended while Execute Query \\\"Execute Query\\\" was materializing its result set.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5c7e9a2d-4b8f-4e1c-9a3d-6f2b8c4e1a7d\",\r\n \"Name\": \"Finance Automation\",\r\n \"Path\": \"Finance Automation\",\r\n \"Description\": \"Folder hosting the finance export and reconciliation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/task.yaml index b19095b734..4fc36738b4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-clr-crash/task.yaml @@ -30,22 +30,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/README.md new file mode 100644 index 0000000000..fa5f850090 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/README.md @@ -0,0 +1,68 @@ +# Execute Query Failure — Failed to Enable Constraints (DataTable schema violation) + +This scenario reproduces a runtime `Execute Query` failure where the **query +executes fine** but the returned rows violate the schema `Execute Query` +auto-infers for the result `DataTable`. Orchestrator surfaces +`Execute Query: Failed to enable constraints. One or more rows contain values +violating non-null, unique, or foreign-key constraints.` +(`System.Data.ConstraintException`). + +## What this scenario uncovers + +**Root Cause:** The `Execute Query 'Customer Orders'` activity's `Sql` is +`SELECT * FROM Customers c INNER JOIN Orders o ON c.CustomerId = o.CustomerId` +— a **one-to-many** join (one customer has many orders). `Execute Query` fills a +`DataTable` whose schema it infers from the source columns, so `CustomerId` +(the primary key of `Customers`) is marked **unique**. The join repeats +`CustomerId` once per order, so `DataTable.EnableConstraints()` throws the +constraint exception. The failure is in the **shape of the result set vs. the +inferred schema**, not in the SQL syntax (the statement parsed and ran). + +The fix is to **reshape the query so the returned key column is unique** — stop +using `SELECT *` and select the explicit columns you need at the intended grain +(e.g. project the order key `OrderId`, which is unique, or drop the repeating +`CustomerId`). `SELECT DISTINCT` does **not** help here (rows differ by order), +and disabling constraint enforcement only hides the schema mismatch. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 8 — DataTable constraint enablement fails). + +## Why the cause is not one-shot from the log + +The error text names the constraint **category** ("non-null, unique, or +foreign-key") but not **which** column or **why**. So the log alone does not +give the diagnosis — the agent must read the workflow `Sql`, recognise the +`SELECT *` over a one-to-many `JOIN`, and connect the repeated `CustomerId` to +the inferred unique constraint. This deliberately mirrors the +`db-execute-query-sql-syntax-error` / `classic-openapp` design so the scenario +exercises the skill rather than being answerable from a single `jobs logs` read. +Plausible wrong turns to avoid: blaming bad data in the database, a connection +problem, a SQL *syntax* error (branch 3), or recommending "set +EnforceConstraints=false" as the fix. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Query whose `Sql` is `SELECT *` over a 1-to-many Customers→Orders JOIN | + +The decisive evidence chain: + +1. `uip or folders list` → the `Reporting` folder. +2. `uip or jobs list` → the single faulted `CustomerOrderReport` job. +3. `uip or jobs get ` → `Info` carries `Failed to enable constraints ...` wrapping `System.Data.ConstraintException` at `DataTable.EnableConstraints`. +4. `uip or jobs logs ` → Trace lines show the query started and then faulted at constraint enablement (no provider syntax error). +5. `process/Main.xaml` → the `Execute Query` `Sql` is the `SELECT *` over the one-to-many `Customers`→`Orders` join. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (branch 8) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) recognise the fault is a `DataTable` constraint violation on the auto-inferred schema, (b) tie it to the `SELECT *` over a one-to-many JOIN repeating the inferred key column, and (c) recommend reshaping the query so the returned key is unique (explicit columns at the correct grain) — not disabling constraints, not blaming DB data, not a syntax fix. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/0185c4bad2.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/0185c4bad2.json deleted file mode 100644 index e49edcbd55..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/0185c4bad2.json +++ /dev/null @@ -1,37 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-18T07:32:10.612Z", - "Message": "CustomerOrderReport execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-18T07:32:11.140Z", - "Message": "Connect to Database execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-18T07:32:11.520Z", - "Message": "Connect to Database execution ended" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-18T07:32:11.560Z", - "Message": "Execute Query 'Customer Orders' execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-18T07:32:12.041Z", - "Message": "Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints." - } - ], - "Pagination": { - "Returned": 5, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/11b5bd11fa.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/11b5bd11fa.json deleted file mode 100644 index 27fb3e97a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/11b5bd11fa.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-18T07:32:12.041Z", - "Message": "Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\r\nSystem.Data.ConstraintException: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-18T07:32:12.060Z", - "Message": "Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query 'Customer Orders'\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/75949d659e.json deleted file mode 100644 index 41a82d1ef1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/75949d659e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64", - "State": "Faulted", - "ProcessName": "CustomerOrderReport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-18T07:32:10.402Z", - "EndTime": "2026-06-18T07:32:12.088Z", - "CreationTime": "2026-06-18T07:32:10.180Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/README.md deleted file mode 100644 index 114bfaf57f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# Execute Query Failure — Failed to Enable Constraints (DataTable schema violation) - -This scenario reproduces a runtime `Execute Query` failure where the **query -executes fine** but the returned rows violate the schema `Execute Query` -auto-infers for the result `DataTable`. Orchestrator surfaces -`Execute Query: Failed to enable constraints. One or more rows contain values -violating non-null, unique, or foreign-key constraints.` -(`System.Data.ConstraintException`). - -## What this scenario uncovers - -**Root Cause:** The `Execute Query 'Customer Orders'` activity's `Sql` is -`SELECT * FROM Customers c INNER JOIN Orders o ON c.CustomerId = o.CustomerId` -— a **one-to-many** join (one customer has many orders). `Execute Query` fills a -`DataTable` whose schema it infers from the source columns, so `CustomerId` -(the primary key of `Customers`) is marked **unique**. The join repeats -`CustomerId` once per order, so `DataTable.EnableConstraints()` throws the -constraint exception. The failure is in the **shape of the result set vs. the -inferred schema**, not in the SQL syntax (the statement parsed and ran). - -The fix is to **reshape the query so the returned key column is unique** — stop -using `SELECT *` and select the explicit columns you need at the intended grain -(e.g. project the order key `OrderId`, which is unique, or drop the repeating -`CustomerId`). `SELECT DISTINCT` does **not** help here (rows differ by order), -and disabling constraint enforcement only hides the schema mismatch. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 8 — DataTable constraint enablement fails). - -## Why the cause is not one-shot from the log - -The error text names the constraint **category** ("non-null, unique, or -foreign-key") but not **which** column or **why**. So the log alone does not -give the diagnosis — the agent must read the workflow `Sql`, recognise the -`SELECT *` over a one-to-many `JOIN`, and connect the repeated `CustomerId` to -the inferred unique constraint. This deliberately mirrors the -`db-execute-query-sql-syntax-error` / `classic-openapp` design so the scenario -exercises the skill rather than being answerable from a single `jobs logs` read. -Plausible wrong turns to avoid: blaming bad data in the database, a connection -problem, a SQL *syntax* error (branch 3), or recommending "set -EnforceConstraints=false" as the fix. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Query whose `Sql` is `SELECT *` over a 1-to-many Customers→Orders JOIN | -| `data/m/r/*.json` | **synthetic** canned `uip` responses authored from the documented branch-8 signature | -| `data/m/r/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list` → the `Reporting` folder. -2. `uip or jobs list` → the single faulted `CustomerOrderReport` job. -3. `uip or jobs get ` → `Info` carries `Failed to enable constraints ...` wrapping `System.Data.ConstraintException` at `DataTable.EnableConstraints`. -4. `uip or jobs logs ` → Trace lines show the query started and then faulted at constraint enablement (no provider syntax error). -5. `process/Main.xaml` → the `Execute Query` `Sql` is the `SELECT *` over the one-to-many `Customers`→`Orders` join. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (branch 8) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) recognise the fault is a `DataTable` constraint violation on the auto-inferred schema, (b) tie it to the `SELECT *` over a one-to-many JOIN repeating the inferred key column, and (c) recommend reshaping the query so the returned key is unique (explicit columns at the correct grain) — not disabling constraints, not blaming DB data, not a syntax fix. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/ee2628e6b6.json deleted file mode 100644 index beff48444e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5c9d2a71-3e64-4b80-9a15-6b8c0d2e4f31", - "Name": "Reporting", - "Path": "Reporting", - "Description": "Folder hosting the customer and order reporting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/fd9921408f.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/fd9921408f.json deleted file mode 100644 index 44a6c767a4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/fd9921408f.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64", - "StartTime": "2026-06-18T07:32:10.402Z", - "EndTime": "2026-06-18T07:32:12.088Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 30, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "6d2a1b3c-4e5f-4a6b-8c7d-9e0f1a2b3c4d", - "Info": "Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\r\n\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query 'Customer Orders'\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints. ---> System.Data.ConstraintException: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\r\n at System.Data.DataTable.EnableConstraints()\r\n at System.Data.DataTable.set_EnforceConstraints(Boolean value)\r\n at System.Data.DataTable.EndLoadData()\r\n at System.Data.Common.DbDataAdapter.FillLoadDataRow(SchemaMapping mapping)\r\n at System.Data.Common.DbDataAdapter.FillFromReader(DataSet dataset, DataTable datatable, String srcTable, DataReaderContainer dataReader, Int32 startRecord, Int32 maxRecords)\r\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, Dictionary`2 parameters, Int32 timeout)\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-06-18T07:32:10.180Z", - "ReleaseName": "CustomerOrderReport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6112034, - "OrganizationUnitId": 7791624, - "FolderKey": "5c9d2a71-3e64-4b80-9a15-6b8c0d2e4f31", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-18T07:32:12.088Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64", - "Id": 671308842 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/manifest.json deleted file mode 100644 index b92f1bdcb0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query branch-8 scenario (job a2f8c3e1, process CustomerOrderReport, folder Reporting). The query executes fine but returns duplicate values in the column the Execute Query DataTable schema infers as the primary/unique key (a one-to-many Customers->Orders JOIN repeats CustomerId), so DataTable.EnableConstraints throws System.Data.ConstraintException 'Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.' The Error/Info surface names the constraint CATEGORY but NOT which column or why - the cause (SELECT * over a 1-to-many JOIN) is only visible in the workflow SQL, so the diagnosis is not one-shot from the log. Mock dispatcher only; not investigation evidence. Matching is token-aware (quotes/--output ignored); first match wins.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache in _cache/." - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs get a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64", - "file": "fd9921408f.json" - }, - { - "match": "or jobs logs a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64 --level Error", - "file": "11b5bd11fa.json" - }, - { - "match": "or jobs logs a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64", - "file": "0185c4bad2.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/uip-fixture.json new file mode 100644 index 0000000000..6b2bd0816e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/data/uip-fixture.json @@ -0,0 +1,64 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-18T07:32:12.041Z\",\r\n \"Message\": \"Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\\r\\nSystem.Data.ConstraintException: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-18T07:32:12.060Z\",\r\n \"Message\": \"Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query 'Customer Orders'\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64\",\r\n \"StartTime\": \"2026-06-18T07:32:10.402Z\",\r\n \"EndTime\": \"2026-06-18T07:32:12.088Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 30,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"6d2a1b3c-4e5f-4a6b-8c7d-9e0f1a2b3c4d\",\r\n \"Info\": \"Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query 'Customer Orders'\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints. ---> System.Data.ConstraintException: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\\r\\n at System.Data.DataTable.EnableConstraints()\\r\\n at System.Data.DataTable.set_EnforceConstraints(Boolean value)\\r\\n at System.Data.DataTable.EndLoadData()\\r\\n at System.Data.Common.DbDataAdapter.FillLoadDataRow(SchemaMapping mapping)\\r\\n at System.Data.Common.DbDataAdapter.FillFromReader(DataSet dataset, DataTable datatable, String srcTable, DataReaderContainer dataReader, Int32 startRecord, Int32 maxRecords)\\r\\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, Dictionary`2 parameters, Int32 timeout)\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-06-18T07:32:10.180Z\",\r\n \"ReleaseName\": \"CustomerOrderReport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6112034,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"5c9d2a71-3e64-4b80-9a15-6b8c0d2e4f31\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-18T07:32:12.088Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64\",\r\n \"Id\": 671308842\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-18T07:32:10.612Z\",\r\n \"Message\": \"CustomerOrderReport execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-18T07:32:11.140Z\",\r\n \"Message\": \"Connect to Database execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-18T07:32:11.520Z\",\r\n \"Message\": \"Connect to Database execution ended\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-18T07:32:11.560Z\",\r\n \"Message\": \"Execute Query 'Customer Orders' execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-18T07:32:12.041Z\",\r\n \"Message\": \"Execute Query: Failed to enable constraints. One or more rows contain values violating non-null, unique, or foreign-key constraints.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 5,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5c9d2a71-3e64-4b80-9a15-6b8c0d2e4f31\",\r\n \"Name\": \"Reporting\",\r\n \"Path\": \"Reporting\",\r\n \"Description\": \"Folder hosting the customer and order reporting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a2f8c3e1-9d47-4b6a-8e02-1c5f7a9b3d64\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerOrderReport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-18T07:32:10.402Z\",\r\n \"EndTime\": \"2026-06-18T07:32:12.088Z\",\r\n \"CreationTime\": \"2026-06-18T07:32:10.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/task.yaml index c0c92d7808..1ccdb852d7 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-enable-constraints/task.yaml @@ -31,22 +31,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/README.md new file mode 100644 index 0000000000..f977ff120c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/README.md @@ -0,0 +1,82 @@ +# Execute Query Failure — Query Text in the Connection-String Property + +This scenario reproduces a runtime `Connect to Database` / `Execute +Query` failure caused by **crossed fields**: the SQL `SELECT` was pasted +into the `Connect to Database` activity's `ConnectionString` property, +and the real connection string was pasted into the `Execute Query` +activity's `Sql` property. When the connect step runs, the +`Microsoft.Data.SqlClient` connection-string parser tries to parse the +`SELECT` as a connection string and throws +`System.ArgumentException: Format of the initialization string does not +conform to specification starting at index 0`, which Orchestrator +surfaces as `Connect to Database: A database error occurred`. + +## What this scenario uncovers + +**Root Cause:** The two field values are swapped between the two +activities. `ConnectionString` holds the SELECT; `Sql` holds the +`Server=...;Database=...;...` connection string. The connection-string +parser rejects the SELECT at index 0 because it is not a `keyword=value;` +list. The fault is a **connection-string parse failure** at the connect +step — before any SQL ever runs. The fix is to **move the SELECT to the +`Sql` property and the connection string to the `ConnectionString` +property** (and balance the quotes). + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 4 — query text in the connection-string property). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — a `Connect to Database` whose `ConnectionString` is a SQL SELECT, feeding an `Execute Query` whose `Sql` is a connection string | + +The decisive evidence chain: + +1. `uip or folders list --output json` -> the `Finance Automation` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` -> the single faulted `CustomerLedgerSync` job. +3. `uip or jobs get --output json` -> `Info` carries `Connect to Database: A database error occurred` wrapping `System.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.` raised in the `Microsoft.Data.SqlClient` connection-string parser at `Connect to Database`. +4. `uip or jobs logs --output json` -> a Trace line showing the connect attempt opening a connection with the connection string set to the `SELECT ...` text. +5. `process/Main.xaml` -> `Connect to Database` `ConnectionString` = the SELECT; `Execute Query` `Sql` = the connection string (fields crossed). + +## How this differs from sibling branches + +| Branch | Signature | Where the fault is | Fix | +|---|---|---|---| +| 1 (null connection) | `NullReferenceException` at the activity | connection wiring / scope | wire/scope the connection | +| 2 (provider mismatch) | `Keyword not supported: ''` (ArgumentException) post-migration | provider/driver after migration | switch provider, fix keyword, install driver | +| 3 (SQL syntax / concatenation) | `A database error occurred` + `Incorrect syntax near ...` | the SQL text sent to the engine | parameterize / fix the SQL | +| **4 (query in connection string) (this)** | **`A database error occurred` + `Format of the initialization string does not conform to specification`** | **crossed fields — SELECT in `ConnectionString`** | **swap SELECT to `Sql`, connection string to `ConnectionString`** | +| 5 (timeout) | `Timeout expired` | DB-side duration | raise `TimeoutMS` / fix the query | +| 6 (CLR crash) | job exit `0xE0434352`, no activity exception | process-level fault | bound result set / update package | +| 7 (wrong activity) | empty/meaningless `DataTable`, or cast error | activity choice vs statement verb | use Execute Non Query for INSERT/UPDATE/DELETE | + +Critically, this is **NOT branch 3**. Branch 3 is a SQL parse failure +(`Incorrect syntax near ...`) raised by the database engine after a +connection opens. Branch 4 is a **connection-string** parse failure +(`Format of the initialization string does not conform to +specification`) raised by the connection-string parser **before** any +SQL is sent and before any connection opens. The agent must distinguish +the two and not read the error as a SQL syntax problem or as "the +database does not exist". + +## Success criteria + +The test scores the **conclusion, not the trajectory**: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (branch 4) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) name the connection-string parse failure (`Format of the initialization string does not conform to specification`), (b) attribute it to the SQL SELECT being in the `ConnectionString` field (crossed fields), and (c) recommend moving the SELECT to `Sql` and the connection string to `ConnectionString` (and balancing quotes). + +## A note on the fixtures + +The fixtures here are **synthetic / hand-authored** from the documented +branch-4 signature in the playbook — not captured from a real +`.local/investigations/` run. Replace them with verbatim captures from a +real investigation before treating this scenario as a strict regression +signal. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/6f22896857.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/6f22896857.json deleted file mode 100644 index ac3c95252c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/6f22896857.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-29T09:14:01.560Z", - "Message": "Connect to Database: A database error occurred.\r\nSystem.ArgumentException: Format of the initialization string does not conform to specification starting at index 0." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-29T09:14:01.575Z", - "Message": "Connect to Database: A database error occurred.\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/8c5b9dae4a.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/8c5b9dae4a.json deleted file mode 100644 index ac0da5f19c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/8c5b9dae4a.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d", - "State": "Faulted", - "ProcessName": "CustomerLedgerSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-29T09:14:00.402Z", - "EndTime": "2026-05-29T09:14:01.623Z", - "CreationTime": "2026-05-29T09:14:00.190Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/README.md deleted file mode 100644 index b2fd8eac15..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/README.md +++ /dev/null @@ -1,81 +0,0 @@ -# Execute Query Failure — Query Text in the Connection-String Property - -This scenario reproduces a runtime `Connect to Database` / `Execute -Query` failure caused by **crossed fields**: the SQL `SELECT` was pasted -into the `Connect to Database` activity's `ConnectionString` property, -and the real connection string was pasted into the `Execute Query` -activity's `Sql` property. When the connect step runs, the -`Microsoft.Data.SqlClient` connection-string parser tries to parse the -`SELECT` as a connection string and throws -`System.ArgumentException: Format of the initialization string does not -conform to specification starting at index 0`, which Orchestrator -surfaces as `Connect to Database: A database error occurred`. - -## What this scenario uncovers - -**Root Cause:** The two field values are swapped between the two -activities. `ConnectionString` holds the SELECT; `Sql` holds the -`Server=...;Database=...;...` connection string. The connection-string -parser rejects the SELECT at index 0 because it is not a `keyword=value;` -list. The fault is a **connection-string parse failure** at the connect -step — before any SQL ever runs. The fix is to **move the SELECT to the -`Sql` property and the connection string to the `ConnectionString` -property** (and balance the quotes). - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 4 — query text in the connection-string property). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — a `Connect to Database` whose `ConnectionString` is a SQL SELECT, feeding an `Execute Query` whose `Sql` is a connection string | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-4 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` -> the `Finance Automation` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` -> the single faulted `CustomerLedgerSync` job. -3. `uip or jobs get --output json` -> `Info` carries `Connect to Database: A database error occurred` wrapping `System.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.` raised in the `Microsoft.Data.SqlClient` connection-string parser at `Connect to Database`. -4. `uip or jobs logs --output json` -> a Trace line showing the connect attempt opening a connection with the connection string set to the `SELECT ...` text. -5. `process/Main.xaml` -> `Connect to Database` `ConnectionString` = the SELECT; `Execute Query` `Sql` = the connection string (fields crossed). - -## How this differs from sibling branches - -| Branch | Signature | Where the fault is | Fix | -|---|---|---|---| -| 1 (null connection) | `NullReferenceException` at the activity | connection wiring / scope | wire/scope the connection | -| 2 (provider mismatch) | `Keyword not supported: ''` (ArgumentException) post-migration | provider/driver after migration | switch provider, fix keyword, install driver | -| 3 (SQL syntax / concatenation) | `A database error occurred` + `Incorrect syntax near ...` | the SQL text sent to the engine | parameterize / fix the SQL | -| **4 (query in connection string) (this)** | **`A database error occurred` + `Format of the initialization string does not conform to specification`** | **crossed fields — SELECT in `ConnectionString`** | **swap SELECT to `Sql`, connection string to `ConnectionString`** | -| 5 (timeout) | `Timeout expired` | DB-side duration | raise `TimeoutMS` / fix the query | -| 6 (CLR crash) | job exit `0xE0434352`, no activity exception | process-level fault | bound result set / update package | -| 7 (wrong activity) | empty/meaningless `DataTable`, or cast error | activity choice vs statement verb | use Execute Non Query for INSERT/UPDATE/DELETE | - -Critically, this is **NOT branch 3**. Branch 3 is a SQL parse failure -(`Incorrect syntax near ...`) raised by the database engine after a -connection opens. Branch 4 is a **connection-string** parse failure -(`Format of the initialization string does not conform to -specification`) raised by the connection-string parser **before** any -SQL is sent and before any connection opens. The agent must distinguish -the two and not read the error as a SQL syntax problem or as "the -database does not exist". - -## Success criteria - -The test scores the **conclusion, not the trajectory**: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (branch 4) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) name the connection-string parse failure (`Format of the initialization string does not conform to specification`), (b) attribute it to the SQL SELECT being in the `ConnectionString` field (crossed fields), and (c) recommend moving the SELECT to `Sql` and the connection string to `ConnectionString` (and balancing quotes). - -## A note on the fixtures - -The fixtures here are **synthetic / hand-authored** from the documented -branch-4 signature in the playbook — not captured from a real -`.local/investigations/` run. Replace them with verbatim captures from a -real investigation before treating this scenario as a strict regression -signal. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/b3456c85e0.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/b3456c85e0.json deleted file mode 100644 index 2f689a5102..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/b3456c85e0.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f", - "Name": "Finance Automation", - "Path": "Finance Automation", - "Description": "Folder hosting the customer ledger and reconciliation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c4f1a983-6b2d-4e57-9a1c-3d8e0f5b7a26", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/bb79fb2331.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/bb79fb2331.json deleted file mode 100644 index 7f6c1e67b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/bb79fb2331.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-29T09:14:00.940Z", - "Message": "CustomerLedgerSync execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-29T09:14:01.480Z", - "Message": "Connect to Database: opening connection with ProviderName 'Microsoft.Data.SqlClient' and connection string 'SELECT CustomerId, Balance, Region FROM CustomerLedger WHERE Region = ''EU'''" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-29T09:14:01.560Z", - "Message": "Connect to Database: A database error occurred.\r\nSystem.ArgumentException: Format of the initialization string does not conform to specification starting at index 0." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-29T09:14:01.575Z", - "Message": "Connect to Database: A database error occurred.\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/cb005d1a8b.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/cb005d1a8b.json deleted file mode 100644 index 1191bed6ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/cb005d1a8b.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d", - "StartTime": "2026-05-29T09:14:00.402Z", - "EndTime": "2026-05-29T09:14:01.623Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Info": "Connect to Database: A database error occurred.\r\n\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Connect to Database: A database error occurred. ---> System.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.\r\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\r\n at System.Data.Common.DbConnectionOptions.ctor(String connectionString, Hashtable synonyms, Boolean useOdbcRules)\r\n at Microsoft.Data.SqlClient.SqlConnectionString.ctor(String connectionString)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-05-29T09:14:00.190Z", - "ReleaseName": "CustomerLedgerSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5042118, - "OrganizationUnitId": 7791624, - "FolderKey": "6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-29T09:14:01.623Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d", - "Id": 671208842 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/manifest.json deleted file mode 100644 index 32d725af84..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "b3456c85e0.json" - }, - { - "match": "or jobs list --folder-key \"6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f\" --state Faulted --output json", - "file": "8c5b9dae4a.json" - }, - { - "match": "or jobs list --folder-key 6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f --state Faulted --output json", - "file": "8c5b9dae4a.json" - }, - { - "match": "or jobs get \"7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d\" --output json", - "file": "cb005d1a8b.json" - }, - { - "match": "or jobs get 7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d --output json", - "file": "cb005d1a8b.json" - }, - { - "match": "or jobs logs \"7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d\" --level Error --output json", - "file": "6f22896857.json" - }, - { - "match": "or jobs logs 7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d --level Error --output json", - "file": "6f22896857.json" - }, - { - "match": "or jobs logs \"7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d\" --output json", - "file": "bb79fb2331.json" - }, - { - "match": "or jobs logs 7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d --output json", - "file": "bb79fb2331.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/uip-fixture.json new file mode 100644 index 0000000000..46e6cbdf38 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerLedgerSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-29T09:14:00.402Z\",\r\n \"EndTime\": \"2026-05-29T09:14:01.623Z\",\r\n \"CreationTime\": \"2026-05-29T09:14:00.190Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T09:14:01.560Z\",\r\n \"Message\": \"Connect to Database: A database error occurred.\\r\\nSystem.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T09:14:01.575Z\",\r\n \"Message\": \"Connect to Database: A database error occurred.\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d\",\r\n \"StartTime\": \"2026-05-29T09:14:00.402Z\",\r\n \"EndTime\": \"2026-05-29T09:14:01.623Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"Connect to Database: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Connect to Database: A database error occurred. ---> System.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.\\r\\n at System.Data.Common.DbConnectionOptions.GetKeyValuePair(String connectionString, Int32 currentPosition, StringBuilder buffer, Boolean useOdbcRules, String& keyname, String& keyvalue)\\r\\n at System.Data.Common.DbConnectionOptions.ctor(String connectionString, Hashtable synonyms, Boolean useOdbcRules)\\r\\n at Microsoft.Data.SqlClient.SqlConnectionString.ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-05-29T09:14:00.190Z\",\r\n \"ReleaseName\": \"CustomerLedgerSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5042118,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-29T09:14:01.623Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d\",\r\n \"Id\": 671208842\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7e2b5d9a-4c1f-4a8e-b3d6-9f0c2a5e7b1d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-29T09:14:00.940Z\",\r\n \"Message\": \"CustomerLedgerSync execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-29T09:14:01.480Z\",\r\n \"Message\": \"Connect to Database: opening connection with ProviderName 'Microsoft.Data.SqlClient' and connection string 'SELECT CustomerId, Balance, Region FROM CustomerLedger WHERE Region = ''EU'''\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T09:14:01.560Z\",\r\n \"Message\": \"Connect to Database: A database error occurred.\\r\\nSystem.ArgumentException: Format of the initialization string does not conform to specification starting at index 0.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T09:14:01.575Z\",\r\n \"Message\": \"Connect to Database: A database error occurred.\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a3d9f1c-2e5b-4d8a-9c7f-1b4e0a6d3c8f\",\r\n \"Name\": \"Finance Automation\",\r\n \"Path\": \"Finance Automation\",\r\n \"Description\": \"Folder hosting the customer ledger and reconciliation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c4f1a983-6b2d-4e57-9a1c-3d8e0f5b7a26\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/task.yaml index fbf5f828dd..501a4ba593 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-in-connection-string/task.yaml @@ -30,22 +30,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/README.md new file mode 100644 index 0000000000..ccdff99dab --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/README.md @@ -0,0 +1,75 @@ +# Execute Query Failure — Null / Out-of-Scope Connection + +This scenario reproduces a runtime `Execute Query` failure caused by a +`DatabaseConnection` variable that is **out of scope** where the query +runs. `Connect to Database` opens the connection inside a nested +`Sequence`, but `Execute Query` sits outside that sequence — so by the +time it executes, the connection variable is `Nothing`. The activity +throws `System.NullReferenceException` ("Object reference not set to an +instance of an object"). + +## What this scenario uncovers + +**Root Cause:** The `Execute Query` activity reads `ExistingDbConnection` +from `dbConnection`, a variable declared on a nested `Sequence` +("Open Connection") that also hosts the `Connect to Database` activity. +Once execution leaves that inner sequence the variable is disposed and +out of scope; `Execute Query`, in the parent sequence, receives a null +connection and faults. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +— **Branch 1 — null / out-of-scope connection** (medium-confidence +playbook). + +> **Why "medium-confidence":** `NullReferenceException` at the activity +> is the verbatim Branch 1 signature, but Branch 1 has several +> sub-causes (connect never ran, wrong/unwired variable, out-of-scope +> variable, disposed handle). The agent must trace the connection +> variable in the workflow source to land on the scope sub-cause. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — `Connect to Database` inside a nested `Sequence`, `Execute Query` outside it referencing the out-of-scope variable | + +The decisive evidence: + +1. The job exception: `System.NullReferenceException: Object reference not set to an instance of an object.` thrown `at ExecuteQuery "Execute Query"`. +2. The job-log Trace ordering: `Connect to Database` ran and opened the connection, the inner `Sequence "Open Connection"` then closed (disposing `dbConnection`), and only afterward did `Execute Query` start and fault. +3. The workflow source (`Main.xaml`): `dbConnection` is declared on the nested `Sequence`, while `Execute Query` lives in the parent `Sequence` — the scope mismatch. + +## How this differs from sibling Execute Query branches + +| Dimension | Branch 1 (this) | Branch 2 | Branch 3 | Branch 5 | Branch 7 | +|---|---|---|---|---|---| +| Failure | null/out-of-scope connection | provider mismatch (post-migration) | SQL syntax / concatenation | command timeout | wrong activity for statement | +| Signature | `NullReferenceException` | `Keyword not supported` | `A database error occurred` (inner syntax) | `Timeout expired` | empty/cast error | +| Fix layer | workflow (variable scope) | provider config | parameterize SQL | `TimeoutMS` / index | swap activity | + +The agent must recommend fixing the **connection variable's scope** in +the workflow — not the SQL, the provider, the timeout, or the activity +type. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the Execute Query failures playbook (Branch 1) AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) identify the `DatabaseConnection` as null / out of scope, (b) attribute it to the connection variable being declared inside the nested `Sequence`, and (c) recommend declaring the variable at the enclosing scope (or moving `Execute Query` into the connect scope) and confirming the `ExistingDbConnection` wiring + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-query-null-connection --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/6c92f03bd0.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/6c92f03bd0.json deleted file mode 100644 index 2191e04648..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/6c92f03bd0.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a", - "Name": "Finance", - "Path": "Finance", - "Description": "Folder for finance automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/8e9d74b4a0.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/8e9d74b4a0.json deleted file mode 100644 index 07e90a9aa7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/8e9d74b4a0.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c", - "State": "Faulted", - "ProcessName": "CustomerQuery", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:14:02.118Z", - "EndTime": "2026-05-21T09:14:02.701Z", - "CreationTime": "2026-05-21T09:14:01.903Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/README.md deleted file mode 100644 index 53c2140d04..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Execute Query Failure — Null / Out-of-Scope Connection - -This scenario reproduces a runtime `Execute Query` failure caused by a -`DatabaseConnection` variable that is **out of scope** where the query -runs. `Connect to Database` opens the connection inside a nested -`Sequence`, but `Execute Query` sits outside that sequence — so by the -time it executes, the connection variable is `Nothing`. The activity -throws `System.NullReferenceException` ("Object reference not set to an -instance of an object"). - -## What this scenario uncovers - -**Root Cause:** The `Execute Query` activity reads `ExistingDbConnection` -from `dbConnection`, a variable declared on a nested `Sequence` -("Open Connection") that also hosts the `Connect to Database` activity. -Once execution leaves that inner sequence the variable is disposed and -out of scope; `Execute Query`, in the parent sequence, receives a null -connection and faults. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -— **Branch 1 — null / out-of-scope connection** (medium-confidence -playbook). - -> **Why "medium-confidence":** `NullReferenceException` at the activity -> is the verbatim Branch 1 signature, but Branch 1 has several -> sub-causes (connect never ran, wrong/unwired variable, out-of-scope -> variable, disposed handle). The agent must trace the connection -> variable in the workflow source to land on the scope sub-cause. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — `Connect to Database` inside a nested `Sequence`, `Execute Query` outside it referencing the out-of-scope variable | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The job exception: `System.NullReferenceException: Object reference not set to an instance of an object.` thrown `at ExecuteQuery "Execute Query"`. -2. The job-log Trace ordering: `Connect to Database` ran and opened the connection, the inner `Sequence "Open Connection"` then closed (disposing `dbConnection`), and only afterward did `Execute Query` start and fault. -3. The workflow source (`Main.xaml`): `dbConnection` is declared on the nested `Sequence`, while `Execute Query` lives in the parent `Sequence` — the scope mismatch. - -## How this differs from sibling Execute Query branches - -| Dimension | Branch 1 (this) | Branch 2 | Branch 3 | Branch 5 | Branch 7 | -|---|---|---|---|---|---| -| Failure | null/out-of-scope connection | provider mismatch (post-migration) | SQL syntax / concatenation | command timeout | wrong activity for statement | -| Signature | `NullReferenceException` | `Keyword not supported` | `A database error occurred` (inner syntax) | `Timeout expired` | empty/cast error | -| Fix layer | workflow (variable scope) | provider config | parameterize SQL | `TimeoutMS` / index | swap activity | - -The agent must recommend fixing the **connection variable's scope** in -the workflow — not the SQL, the provider, the timeout, or the activity -type. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the Execute Query failures playbook (Branch 1) AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) identify the `DatabaseConnection` as null / out of scope, (b) attribute it to the connection variable being declared inside the nested `Sequence`, and (c) recommend declaring the variable at the enclosing scope (or moving `Execute Query` into the connect scope) and confirming the `ExistingDbConnection` wiring - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-query-null-connection --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/b4c343d544.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/b4c343d544.json deleted file mode 100644 index 27edc0d9f3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/b4c343d544.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c", - "StartTime": "2026-05-21T09:14:02.118Z", - "EndTime": "2026-05-21T09:14:02.701Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7a8b9c0d-1e2f-4a3b-8c5d-6e7f8a9b0c1d", - "Info": "Execute Query: Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Database.Activities.ExecuteQuery.", - "CreationTime": "2026-05-21T09:14:01.903Z", - "ReleaseName": "CustomerQuery", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5041902, - "OrganizationUnitId": 7791624, - "FolderKey": "a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T09:14:02.701Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c", - "Id": 671058913 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/c645697ad3.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/c645697ad3.json deleted file mode 100644 index b38141cda3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/c645697ad3.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Trace", - "TimeStamp": "2026-05-21T09:14:02.401Z", - "Message": "Connect to Database: execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-21T09:14:02.512Z", - "Message": "Connect to Database: execution ended (connection opened; assigned to variable 'dbConnection' scoped to Sequence 'Open Connection')" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-21T09:14:02.540Z", - "Message": "Sequence 'Open Connection': execution ended (scope closed; 'dbConnection' disposed and out of scope)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-21T09:14:02.585Z", - "Message": "Execute Query: execution started (ExistingDbConnection bound to 'dbConnection')" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:02.640Z", - "Message": "Execute Query: Object reference not set to an instance of an object." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:02.690Z", - "Message": "System.NullReferenceException: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/manifest.json deleted file mode 100644 index 8a69f38abc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6c92f03bd0.json" - }, - { - "match": "or jobs list --folder-key \"a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a\" --state Faulted --output json", - "file": "8e9d74b4a0.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a --state Faulted --output json", - "file": "8e9d74b4a0.json" - }, - { - "match": "or jobs get \"e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c\" --output json", - "file": "b4c343d544.json" - }, - { - "match": "or jobs get e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c --output json", - "file": "b4c343d544.json" - }, - { - "match": "or jobs logs \"e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c\" --level Error --output json", - "file": "c645697ad3.json" - }, - { - "match": "or jobs logs e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c --level Error --output json", - "file": "c645697ad3.json" - }, - { - "match": "or jobs logs \"e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c\" --output json", - "file": "c645697ad3.json" - }, - { - "match": "or jobs logs e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c --output json", - "file": "c645697ad3.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/uip-fixture.json new file mode 100644 index 0000000000..45bb96b3ec --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerQuery\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:14:02.118Z\",\r\n \"EndTime\": \"2026-05-21T09:14:02.701Z\",\r\n \"CreationTime\": \"2026-05-21T09:14:01.903Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.401Z\",\r\n \"Message\": \"Connect to Database: execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.512Z\",\r\n \"Message\": \"Connect to Database: execution ended (connection opened; assigned to variable 'dbConnection' scoped to Sequence 'Open Connection')\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.540Z\",\r\n \"Message\": \"Sequence 'Open Connection': execution ended (scope closed; 'dbConnection' disposed and out of scope)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.585Z\",\r\n \"Message\": \"Execute Query: execution started (ExistingDbConnection bound to 'dbConnection')\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.640Z\",\r\n \"Message\": \"Execute Query: Object reference not set to an instance of an object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.690Z\",\r\n \"Message\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"StartTime\": \"2026-05-21T09:14:02.118Z\",\r\n \"EndTime\": \"2026-05-21T09:14:02.701Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7a8b9c0d-1e2f-4a3b-8c5d-6e7f8a9b0c1d\",\r\n \"Info\": \"Execute Query: Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Database.Activities.ExecuteQuery.\",\r\n \"CreationTime\": \"2026-05-21T09:14:01.903Z\",\r\n \"ReleaseName\": \"CustomerQuery\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5041902,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T09:14:02.701Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"Id\": 671058913\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e1f2a3b4-1f5d-4e8c-9b6a-2d3e4f5a6b7c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.401Z\",\r\n \"Message\": \"Connect to Database: execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.512Z\",\r\n \"Message\": \"Connect to Database: execution ended (connection opened; assigned to variable 'dbConnection' scoped to Sequence 'Open Connection')\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.540Z\",\r\n \"Message\": \"Sequence 'Open Connection': execution ended (scope closed; 'dbConnection' disposed and out of scope)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.585Z\",\r\n \"Message\": \"Execute Query: execution started (ExistingDbConnection bound to 'dbConnection')\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.640Z\",\r\n \"Message\": \"Execute Query: Object reference not set to an instance of an object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.690Z\",\r\n \"Message\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-2e8c-4f6b-9d5a-1c2b3d4e5f6a\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"Folder for finance automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/task.yaml index fdf3858ed9..8fe838b7a4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-null-connection/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/README.md new file mode 100644 index 0000000000..0fd51c3b86 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/README.md @@ -0,0 +1,77 @@ +# Execute Query Failure — Driver/Provider Mismatch After Migration + +This scenario reproduces a runtime `Execute Query` failure that begins +right after a project is upgraded from **Windows - Legacy** +(.NET Framework) to **Windows** (.NET 6+). The `Connect to Database` +activity still uses `ProviderName="System.Data.SqlClient"` with a +connection string carrying a keyword the modern provider rejects +(`Provider=SQLOLEDB;...`). The modern connection-string parser +(`Microsoft.Data.SqlClient`) throws +`System.ArgumentException: Keyword not supported: 'Provider'`, which +Orchestrator surfaces as `Execute Query: A database error occurred`. + +## What this scenario uncovers + +**Root Cause:** The project's `targetFramework` is `Windows` (migrated), +but the Database activity was never updated off the legacy provider. On +.NET 6+ the `System.Data.SqlClient` / OLE-DB keyword combination is +invalid; `Microsoft.Data.SqlClient` rejects `Provider=SQLOLEDB`. The fix +is to **switch the provider to `Microsoft.Data.SqlClient`**, drop/adjust +the rejected connection-string keyword(s), and confirm the driver is +installed on the Robot host. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 2 — driver/provider mismatch after migration). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — `targetFramework: "Windows"`, a `Connect to Database` using `System.Data.SqlClient` with a `Provider=SQLOLEDB;...` connection string feeding an `Execute Query` | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Finance Automation` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `InvoiceReconciliation` job. +3. `uip or jobs get --output json` → `Info` carries `Execute Query: A database error occurred` wrapping `System.ArgumentException: Keyword not supported: 'Provider'.` raised in the `Microsoft.Data.SqlClient` connection-string parser at `Connect to Database`. +4. `uip or jobs logs --output json` → a Trace line showing the connect attempt with `ProviderName 'System.Data.SqlClient'` and the `Provider=SQLOLEDB;...` connection string. +5. `process/project.json` → `targetFramework: "Windows"` (migrated); `process/Main.xaml` → `Connect to Database` with `ProviderName="System.Data.SqlClient"` and the offending connection string. + +## How this differs from sibling branches + +| Dimension | branch 1 (null connection) | branch 2 (provider mismatch) (this) | branch 3 (concatenation) | branch 5 (timeout) | +|---|---|---|---|---| +| Connection opens? | no | **no — parser rejects keyword** | yes | yes | +| Error signature | `NullReferenceException` | **`Keyword not supported: 'Provider'` (ArgumentException)** | `A database error occurred` + `Incorrect syntax near ...` | `Timeout expired` | +| Where the fault is | wiring/scope | **provider/driver after migration** | the SQL text | DB-side duration | +| Fix | wire/scope the connection | **switch to Microsoft.Data.SqlClient + fix keyword + install driver** | parameterize the query | raise `TimeoutMS` / fix the query | + +The failing layer here is the **provider/connection-string +configuration after migration**, not the connection wiring, the SQL +text, a timeout, or the database. The agent must recommend switching to +`Microsoft.Data.SqlClient` and adjusting the rejected keyword(s) — not +blame a null connection, SQL syntax, a timeout, a CLR crash, the wrong +activity, or a DB outage. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (branch 2) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) name the `Keyword not supported` ArgumentException from the connection-string parser, (b) attribute it to the Windows migration while the activity still uses `System.Data.SqlClient` with a rejected keyword, and (c) recommend switching to `Microsoft.Data.SqlClient`, adjusting the keyword(s), and confirming the driver is installed on the Robot host. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-query-provider-mismatch --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/1e6444dce7.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/1e6444dce7.json deleted file mode 100644 index cfd8c00197..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/1e6444dce7.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e", - "Name": "Finance Automation", - "Path": "Finance Automation", - "Description": "Folder hosting the invoice reconciliation and reporting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c4f1a983-6b2d-4e57-9a1c-3d8e0f5b7a26", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/71743e9325.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/71743e9325.json deleted file mode 100644 index 2eff968740..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/71743e9325.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-28T07:32:12.804Z", - "Message": "Execute Query: A database error occurred.\r\nSystem.ArgumentException: Keyword not supported: 'Provider'.\r\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(String keyword)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-28T07:32:12.819Z", - "Message": "Execute Query: A database error occurred.\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/8337f8aea4.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/8337f8aea4.json deleted file mode 100644 index 95b39b49f0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/8337f8aea4.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f", - "StartTime": "2026-05-28T07:32:11.402Z", - "EndTime": "2026-05-28T07:32:12.871Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Info": "Execute Query: A database error occurred.\r\n\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Execute Query: A database error occurred. ---> System.ArgumentException: Keyword not supported: 'Provider'.\r\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(String keyword)\r\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.set_Item(String keyword, Object value)\r\n at Microsoft.Data.SqlClient.SqlConnectionString.ctor(String connectionString)\r\n at Microsoft.Data.SqlClient.SqlConnection.ConnectionString_Set(DbConnectionPoolKey key)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-05-28T07:32:11.190Z", - "ReleaseName": "InvoiceReconciliation", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5042118, - "OrganizationUnitId": 7791624, - "FolderKey": "5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-28T07:32:12.871Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f", - "Id": 671208842 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/README.md deleted file mode 100644 index 46c1663f08..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/README.md +++ /dev/null @@ -1,76 +0,0 @@ -# Execute Query Failure — Driver/Provider Mismatch After Migration - -This scenario reproduces a runtime `Execute Query` failure that begins -right after a project is upgraded from **Windows - Legacy** -(.NET Framework) to **Windows** (.NET 6+). The `Connect to Database` -activity still uses `ProviderName="System.Data.SqlClient"` with a -connection string carrying a keyword the modern provider rejects -(`Provider=SQLOLEDB;...`). The modern connection-string parser -(`Microsoft.Data.SqlClient`) throws -`System.ArgumentException: Keyword not supported: 'Provider'`, which -Orchestrator surfaces as `Execute Query: A database error occurred`. - -## What this scenario uncovers - -**Root Cause:** The project's `targetFramework` is `Windows` (migrated), -but the Database activity was never updated off the legacy provider. On -.NET 6+ the `System.Data.SqlClient` / OLE-DB keyword combination is -invalid; `Microsoft.Data.SqlClient` rejects `Provider=SQLOLEDB`. The fix -is to **switch the provider to `Microsoft.Data.SqlClient`**, drop/adjust -the rejected connection-string keyword(s), and confirm the driver is -installed on the Robot host. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 2 — driver/provider mismatch after migration). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — `targetFramework: "Windows"`, a `Connect to Database` using `System.Data.SqlClient` with a `Provider=SQLOLEDB;...` connection string feeding an `Execute Query` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Finance Automation` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `InvoiceReconciliation` job. -3. `uip or jobs get --output json` → `Info` carries `Execute Query: A database error occurred` wrapping `System.ArgumentException: Keyword not supported: 'Provider'.` raised in the `Microsoft.Data.SqlClient` connection-string parser at `Connect to Database`. -4. `uip or jobs logs --output json` → a Trace line showing the connect attempt with `ProviderName 'System.Data.SqlClient'` and the `Provider=SQLOLEDB;...` connection string. -5. `process/project.json` → `targetFramework: "Windows"` (migrated); `process/Main.xaml` → `Connect to Database` with `ProviderName="System.Data.SqlClient"` and the offending connection string. - -## How this differs from sibling branches - -| Dimension | branch 1 (null connection) | branch 2 (provider mismatch) (this) | branch 3 (concatenation) | branch 5 (timeout) | -|---|---|---|---|---| -| Connection opens? | no | **no — parser rejects keyword** | yes | yes | -| Error signature | `NullReferenceException` | **`Keyword not supported: 'Provider'` (ArgumentException)** | `A database error occurred` + `Incorrect syntax near ...` | `Timeout expired` | -| Where the fault is | wiring/scope | **provider/driver after migration** | the SQL text | DB-side duration | -| Fix | wire/scope the connection | **switch to Microsoft.Data.SqlClient + fix keyword + install driver** | parameterize the query | raise `TimeoutMS` / fix the query | - -The failing layer here is the **provider/connection-string -configuration after migration**, not the connection wiring, the SQL -text, a timeout, or the database. The agent must recommend switching to -`Microsoft.Data.SqlClient` and adjusting the rejected keyword(s) — not -blame a null connection, SQL syntax, a timeout, a CLR crash, the wrong -activity, or a DB outage. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (branch 2) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) name the `Keyword not supported` ArgumentException from the connection-string parser, (b) attribute it to the Windows migration while the activity still uses `System.Data.SqlClient` with a rejected keyword, and (c) recommend switching to `Microsoft.Data.SqlClient`, adjusting the keyword(s), and confirming the driver is installed on the Robot host. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-query-provider-mismatch --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/b46ba6588a.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/b46ba6588a.json deleted file mode 100644 index d14949b329..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/b46ba6588a.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f", - "State": "Faulted", - "ProcessName": "InvoiceReconciliation", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T07:32:11.402Z", - "EndTime": "2026-05-28T07:32:12.871Z", - "CreationTime": "2026-05-28T07:32:11.190Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/dcdb016e32.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/dcdb016e32.json deleted file mode 100644 index 0a5c5fd2ba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/dcdb016e32.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-28T07:32:12.140Z", - "Message": "InvoiceReconciliation execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-28T07:32:12.702Z", - "Message": "Connect to Database: opening connection with ProviderName 'System.Data.SqlClient' and connection string 'Provider=SQLOLEDB;Server=sqlprod02;Database=Finance;Integrated Security=SSPI;Encrypt=true'" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-28T07:32:12.804Z", - "Message": "Execute Query: A database error occurred.\r\nSystem.ArgumentException: Keyword not supported: 'Provider'.\r\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(String keyword)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-28T07:32:12.819Z", - "Message": "Execute Query: A database error occurred.\r\nin Main.xaml\r\n at DatabaseConnect \"Connect to Database\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/manifest.json deleted file mode 100644 index 5568a5d44d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "1e6444dce7.json" - }, - { - "match": "or jobs list --folder-key \"5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e\" --state Faulted --output json", - "file": "b46ba6588a.json" - }, - { - "match": "or jobs list --folder-key 5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e --state Faulted --output json", - "file": "b46ba6588a.json" - }, - { - "match": "or jobs get \"4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f\" --output json", - "file": "8337f8aea4.json" - }, - { - "match": "or jobs get 4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f --output json", - "file": "8337f8aea4.json" - }, - { - "match": "or jobs logs \"4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f\" --level Error --output json", - "file": "71743e9325.json" - }, - { - "match": "or jobs logs 4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f --level Error --output json", - "file": "71743e9325.json" - }, - { - "match": "or jobs logs \"4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f\" --output json", - "file": "dcdb016e32.json" - }, - { - "match": "or jobs logs 4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f --output json", - "file": "dcdb016e32.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..075ecfd1b7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceReconciliation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T07:32:11.402Z\",\r\n \"EndTime\": \"2026-05-28T07:32:12.871Z\",\r\n \"CreationTime\": \"2026-05-28T07:32:11.190Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-28T07:32:12.804Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nSystem.ArgumentException: Keyword not supported: 'Provider'.\\r\\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(String keyword)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-28T07:32:12.819Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f\",\r\n \"StartTime\": \"2026-05-28T07:32:11.402Z\",\r\n \"EndTime\": \"2026-05-28T07:32:12.871Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"Execute Query: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Execute Query: A database error occurred. ---> System.ArgumentException: Keyword not supported: 'Provider'.\\r\\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(String keyword)\\r\\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.set_Item(String keyword, Object value)\\r\\n at Microsoft.Data.SqlClient.SqlConnectionString.ctor(String connectionString)\\r\\n at Microsoft.Data.SqlClient.SqlConnection.ConnectionString_Set(DbConnectionPoolKey key)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-05-28T07:32:11.190Z\",\r\n \"ReleaseName\": \"InvoiceReconciliation\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5042118,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-28T07:32:12.871Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f\",\r\n \"Id\": 671208842\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4d8a1c3f-9b2e-4c7a-8d1f-6e3a5b0c2d4f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-28T07:32:12.140Z\",\r\n \"Message\": \"InvoiceReconciliation execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-28T07:32:12.702Z\",\r\n \"Message\": \"Connect to Database: opening connection with ProviderName 'System.Data.SqlClient' and connection string 'Provider=SQLOLEDB;Server=sqlprod02;Database=Finance;Integrated Security=SSPI;Encrypt=true'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-28T07:32:12.804Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nSystem.ArgumentException: Keyword not supported: 'Provider'.\\r\\n at Microsoft.Data.SqlClient.SqlConnectionStringBuilder.GetIndex(String keyword)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-28T07:32:12.819Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nin Main.xaml\\r\\n at DatabaseConnect \\\"Connect to Database\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5e2c9a7b-1d4f-4a6c-b3e8-7f0a2c5d9b1e\",\r\n \"Name\": \"Finance Automation\",\r\n \"Path\": \"Finance Automation\",\r\n \"Description\": \"Folder hosting the invoice reconciliation and reporting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c4f1a983-6b2d-4e57-9a1c-3d8e0f5b7a26\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/task.yaml index 87ce728051..ed31d75b4e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-provider-mismatch/task.yaml @@ -30,22 +30,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/README.md new file mode 100644 index 0000000000..149cd60eb7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/README.md @@ -0,0 +1,75 @@ +# Execute Query Failure — SQL Syntax Error from Unsafe Concatenation + +This scenario reproduces a runtime `Execute Query` failure caused by +**building SQL with string concatenation**. The activity concatenates a +UiPath variable straight into the statement; at runtime the variable +holds a value with an apostrophe (`O'Brien`), the apostrophe closes the +SQL string literal early, and the provider rejects the malformed +statement. Orchestrator surfaces `Execute Query: A database error +occurred` wrapping a `Microsoft.Data.SqlClient` `SqlException`: +`Incorrect syntax near 'Brien'.` + +## What this scenario uncovers + +**Root Cause:** The `Execute Query` activity's `Sql` is built as +`"SELECT * FROM Customers WHERE LastName = '" + in_LastName + "'"`. +With `in_LastName = "O'Brien"` the resolved SQL becomes +`SELECT * FROM Customers WHERE LastName = 'O'Brien'` — the embedded +apostrophe breaks the literal, so SQL Server fails to parse it. The fix +is to **parameterize**: use a static `SELECT ... WHERE LastName = +@lastName` and map `in_LastName` to `@lastName` via the activity's +Parameters collection. This also closes the SQL-injection risk. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 3 — SQL syntax error / unsafe string concatenation). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Query whose `Sql` is a concatenation expression, behind a valid Connect/Execute structure | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Sales Automation` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `CustomerLookup` job. +3. `uip or jobs get --output json` → `Info` carries `Execute Query: A database error occurred` wrapping `Incorrect syntax near 'Brien'. Unclosed quotation mark after the character string ''.` +4. `uip or jobs logs --output json` → a Trace line showing the **resolved** SQL `... LastName = 'O'Brien'` — the smoking gun proving the apostrophe broke the literal. +5. `process/Main.xaml` → the `Execute Query` `Sql` is the concatenation expression `"...'" + in_LastName + "'"` with `in_LastName` defaulting to `O'Brien`. + +## How this differs from sibling branches + +| Dimension | branch 1 (null connection) | branch 2 (provider mismatch) | branch 3 (concatenation) (this) | branch 5 (timeout) | +|---|---|---|---|---| +| Connection opens? | no | no/garbled | **yes** | yes | +| Error signature | `NullReferenceException` | `Keyword not supported` | **`A database error occurred` + `Incorrect syntax near ...`** | `Timeout expired` | +| Where the fault is | wiring/scope | provider/driver | **the SQL text** | DB-side duration | +| Fix | wire/scope the connection | switch provider | **parameterize the query** | raise `TimeoutMS` / fix the query | + +The failing layer here is the **statement text**, not the connection, +the driver, the table, or a timeout. The agent must recommend +parameterizing with a named `@parameter` — not escaping the quote, not +blaming the database, not adjusting a timeout. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (branch 3) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) recognize the SQL is built by concatenating `in_LastName`, (b) explain the apostrophe in `O'Brien` breaks the string literal, and (c) recommend parameterizing with a named `@parameter` via the Parameters collection. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-query-sql-syntax-error --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/00a3d8e264.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/00a3d8e264.json deleted file mode 100644 index ddc0408908..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/00a3d8e264.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:03.601Z", - "Message": "Execute Query: A database error occurred.\r\nMicrosoft.Data.SqlClient.SqlException: Incorrect syntax near 'Brien'.\r\nUnclosed quotation mark after the character string ''." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:03.618Z", - "Message": "Execute Query: A database error occurred.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/1f899ce981.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/1f899ce981.json deleted file mode 100644 index a72106f3e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/1f899ce981.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e", - "StartTime": "2026-05-21T09:14:02.118Z", - "EndTime": "2026-05-21T09:14:03.642Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7c1d2e3f-4a5b-4c6d-9e8f-1a2b3c4d5e6f", - "Info": "Execute Query: A database error occurred.\r\n\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Execute Query: A database error occurred. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Incorrect syntax near 'Brien'.\r\nUnclosed quotation mark after the character string ''.\r\n at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()\r\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, Dictionary`2 parameters, Int32 timeout)\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-05-21T09:14:01.905Z", - "ReleaseName": "CustomerLookup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5041992, - "OrganizationUnitId": 7791624, - "FolderKey": "3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T09:14:03.642Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e", - "Id": 671204418 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/42caa3faf9.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/42caa3faf9.json deleted file mode 100644 index 654094fdbc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/42caa3faf9.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-21T09:14:02.940Z", - "Message": "CustomerLookup execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-21T09:14:03.502Z", - "Message": "Execute Query: executing SQL against connection 'Server=sqlprod01;Database=Sales': SELECT * FROM Customers WHERE LastName = 'O'Brien'" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:03.601Z", - "Message": "Execute Query: A database error occurred.\r\nMicrosoft.Data.SqlClient.SqlException: Incorrect syntax near 'Brien'.\r\nUnclosed quotation mark after the character string ''." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-21T09:14:03.618Z", - "Message": "Execute Query: A database error occurred.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/9e8289fb14.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/9e8289fb14.json deleted file mode 100644 index 48485a2852..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/9e8289fb14.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e", - "State": "Faulted", - "ProcessName": "CustomerLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:14:02.118Z", - "EndTime": "2026-05-21T09:14:03.642Z", - "CreationTime": "2026-05-21T09:14:01.905Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/README.md deleted file mode 100644 index 576dd789b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Execute Query Failure — SQL Syntax Error from Unsafe Concatenation - -This scenario reproduces a runtime `Execute Query` failure caused by -**building SQL with string concatenation**. The activity concatenates a -UiPath variable straight into the statement; at runtime the variable -holds a value with an apostrophe (`O'Brien`), the apostrophe closes the -SQL string literal early, and the provider rejects the malformed -statement. Orchestrator surfaces `Execute Query: A database error -occurred` wrapping a `Microsoft.Data.SqlClient` `SqlException`: -`Incorrect syntax near 'Brien'.` - -## What this scenario uncovers - -**Root Cause:** The `Execute Query` activity's `Sql` is built as -`"SELECT * FROM Customers WHERE LastName = '" + in_LastName + "'"`. -With `in_LastName = "O'Brien"` the resolved SQL becomes -`SELECT * FROM Customers WHERE LastName = 'O'Brien'` — the embedded -apostrophe breaks the literal, so SQL Server fails to parse it. The fix -is to **parameterize**: use a static `SELECT ... WHERE LastName = -@lastName` and map `in_LastName` to `@lastName` via the activity's -Parameters collection. This also closes the SQL-injection risk. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 3 — SQL syntax error / unsafe string concatenation). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Query whose `Sql` is a concatenation expression, behind a valid Connect/Execute structure | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-3 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Sales Automation` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `CustomerLookup` job. -3. `uip or jobs get --output json` → `Info` carries `Execute Query: A database error occurred` wrapping `Incorrect syntax near 'Brien'. Unclosed quotation mark after the character string ''.` -4. `uip or jobs logs --output json` → a Trace line showing the **resolved** SQL `... LastName = 'O'Brien'` — the smoking gun proving the apostrophe broke the literal. -5. `process/Main.xaml` → the `Execute Query` `Sql` is the concatenation expression `"...'" + in_LastName + "'"` with `in_LastName` defaulting to `O'Brien`. - -## How this differs from sibling branches - -| Dimension | branch 1 (null connection) | branch 2 (provider mismatch) | branch 3 (concatenation) (this) | branch 5 (timeout) | -|---|---|---|---|---| -| Connection opens? | no | no/garbled | **yes** | yes | -| Error signature | `NullReferenceException` | `Keyword not supported` | **`A database error occurred` + `Incorrect syntax near ...`** | `Timeout expired` | -| Where the fault is | wiring/scope | provider/driver | **the SQL text** | DB-side duration | -| Fix | wire/scope the connection | switch provider | **parameterize the query** | raise `TimeoutMS` / fix the query | - -The failing layer here is the **statement text**, not the connection, -the driver, the table, or a timeout. The agent must recommend -parameterizing with a named `@parameter` — not escaping the quote, not -blaming the database, not adjusting a timeout. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (branch 3) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) recognize the SQL is built by concatenating `in_LastName`, (b) explain the apostrophe in `O'Brien` breaks the string literal, and (c) recommend parameterizing with a named `@parameter` via the Parameters collection. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-query-sql-syntax-error --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/a47b967d5c.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/a47b967d5c.json deleted file mode 100644 index 1ff102ebd7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/a47b967d5c.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d", - "Name": "Sales Automation", - "Path": "Sales Automation", - "Description": "Folder hosting the customer lookup and reporting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/manifest.json deleted file mode 100644 index fd2b56ad0a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "a47b967d5c.json" - }, - { - "match": "or jobs list --folder-key \"3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d\" --state Faulted --output json", - "file": "9e8289fb14.json" - }, - { - "match": "or jobs list --folder-key 3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d --state Faulted --output json", - "file": "9e8289fb14.json" - }, - { - "match": "or jobs get \"8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e\" --output json", - "file": "1f899ce981.json" - }, - { - "match": "or jobs get 8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e --output json", - "file": "1f899ce981.json" - }, - { - "match": "or jobs logs \"8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e\" --level Error --output json", - "file": "00a3d8e264.json" - }, - { - "match": "or jobs logs 8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e --level Error --output json", - "file": "00a3d8e264.json" - }, - { - "match": "or jobs logs \"8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e\" --output json", - "file": "42caa3faf9.json" - }, - { - "match": "or jobs logs 8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e --output json", - "file": "42caa3faf9.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/uip-fixture.json new file mode 100644 index 0000000000..62af0e01fa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:14:02.118Z\",\r\n \"EndTime\": \"2026-05-21T09:14:03.642Z\",\r\n \"CreationTime\": \"2026-05-21T09:14:01.905Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.601Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nMicrosoft.Data.SqlClient.SqlException: Incorrect syntax near 'Brien'.\\r\\nUnclosed quotation mark after the character string ''.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.618Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e\",\r\n \"StartTime\": \"2026-05-21T09:14:02.118Z\",\r\n \"EndTime\": \"2026-05-21T09:14:03.642Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7c1d2e3f-4a5b-4c6d-9e8f-1a2b3c4d5e6f\",\r\n \"Info\": \"Execute Query: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Execute Query: A database error occurred. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Incorrect syntax near 'Brien'.\\r\\nUnclosed quotation mark after the character string ''.\\r\\n at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()\\r\\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, Dictionary`2 parameters, Int32 timeout)\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-05-21T09:14:01.905Z\",\r\n \"ReleaseName\": \"CustomerLookup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5041992,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T09:14:03.642Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e\",\r\n \"Id\": 671204418\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8b4d2f1a-6c3e-4a7b-9d8e-1f2a3b4c5d6e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-21T09:14:02.940Z\",\r\n \"Message\": \"CustomerLookup execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.502Z\",\r\n \"Message\": \"Execute Query: executing SQL against connection 'Server=sqlprod01;Database=Sales': SELECT * FROM Customers WHERE LastName = 'O'Brien'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.601Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nMicrosoft.Data.SqlClient.SqlException: Incorrect syntax near 'Brien'.\\r\\nUnclosed quotation mark after the character string ''.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T09:14:03.618Z\",\r\n \"Message\": \"Execute Query: A database error occurred.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1f6c8e-7b2d-4e9a-8c1f-5d6e7a8b9c0d\",\r\n \"Name\": \"Sales Automation\",\r\n \"Path\": \"Sales Automation\",\r\n \"Description\": \"Folder hosting the customer lookup and reporting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/task.yaml index 718c4f0b3a..5982f16621 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-sql-syntax-error/task.yaml @@ -28,22 +28,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/README.md new file mode 100644 index 0000000000..07d264d33e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/README.md @@ -0,0 +1,79 @@ +# Execute Query Failure — Command Timeout Exceeded (Runaway Un-indexed Query) + +This scenario reproduces a runtime `Execute Query` failure caused by a +**query that exceeds the activity's `TimeoutMS`**. The activity runs an +expensive, un-indexed statement; the provider performs a full-table scan, +runs past the timeout, and aborts. Orchestrator surfaces +`Execute Query: Timeout expired. The timeout period elapsed prior to +completion of the operation or the server is not responding.` + +## What this scenario uncovers + +**Root Cause:** The `Execute Query` activity's `Sql` is +`SELECT * FROM Orders WHERE UPPER(Notes) LIKE '%...%'`. `UPPER(Notes)` +under a leading-wildcard `LIKE` is not sargable and `Orders` has no index +on `Notes`, so the provider does a full-table scan. `TimeoutMS` is left at +the default `30000` (30 s, **milliseconds**); the scan runs past 30 s and +the provider aborts with `Timeout expired`. The fix has two tiers: the +**real fix** is fixing the query/index (sargable predicate + supporting +index, narrower result set); raising `TimeoutMS` (in milliseconds) is only +a **stopgap** because it just delays the same failure on a runaway query. + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 5 — command timeout exceeded). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Execute Query with default `TimeoutMS` running an un-indexed full-scan query behind a valid Connect/Execute structure | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Sales Automation` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `OrderNotesSearch` job (ran ~30 s ≈ `TimeoutMS`). +3. `uip or jobs get --output json` → `Info` carries `Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.` +4. `uip or jobs logs --output json` → Trace lines showing the connection opened, the query started with `TimeoutMS=30000`, then `command aborted after 30000 ms ... full-table scan` — the smoking gun proving the query duration exceeded the timeout. +5. `process/Main.xaml` → the `Execute Query` `Sql` is the un-indexed `UPPER(Notes) LIKE '%...%'` scan with `TimeoutMS="30000"` (default). + +## How this differs from sibling branches + +| Dimension | branch 1 (null connection) | branch 3 (concatenation) | branch 5 (timeout) (this) | +|---|---|---|---| +| Connection opens? | no | yes | **yes** | +| SQL valid? | n/a | no (malformed) | **yes (just slow)** | +| Error signature | `NullReferenceException` | `A database error occurred` + `Incorrect syntax near ...` | **`Timeout expired ...`** | +| Where the fault is | wiring/scope | the SQL text | **query DURATION vs `TimeoutMS`** | +| Fix | wire/scope the connection | parameterize the query | **fix the index/query (real); raise `TimeoutMS` (stopgap)** | + +The failing layer here is the **query duration exceeding `TimeoutMS`**, +not the connection, the driver, the statement text, or a DB outage. The +agent must recognize this is a runaway un-indexed query — so the real fix +is the index/query, and raising `TimeoutMS` (in milliseconds) is only a +temporary stopgap. It must not blame a null connection, SQL syntax, a +provider mismatch, a CLR crash, the wrong activity, or "the database is +down". + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (Branch 5) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) name the `Timeout expired` failure at Execute Query, (b) attribute it to the query running longer than `TimeoutMS` because it is a runaway un-indexed full-table scan, and (c) recommend the real fix = add an index / narrow the query, noting raising `TimeoutMS` (milliseconds) is only a stopgap. +- Partial credit (0.8) if the agent matches the timeout branch but treats it as a legitimately long query and recommends only raising `TimeoutMS`. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-query-timeout-expired --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/357f120a5c.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/357f120a5c.json deleted file mode 100644 index fb1d36427e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/357f120a5c.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-22T11:02:08.940Z", - "Message": "OrderNotesSearch execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-22T11:02:09.011Z", - "Message": "Connect to Database: connection opened to 'Server=sqlprod01;Database=Sales' using provider Microsoft.Data.SqlClient" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-22T11:02:09.114Z", - "Message": "Execute Query: TimeoutMS=30000; executing SQL against connection 'Server=sqlprod01;Database=Sales': SELECT * FROM Orders WHERE UPPER(Notes) LIKE '%URGENT REFUND%'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-22T11:02:38.640Z", - "Message": "Execute Query: command aborted after 30000 ms; the query did not complete within TimeoutMS. Orders has no index on Notes and UPPER(Notes) is not sargable, so the provider performed a full-table scan." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-22T11:02:38.689Z", - "Message": "Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\nMicrosoft.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-22T11:02:38.712Z", - "Message": "Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/3e6aa765dd.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/3e6aa765dd.json deleted file mode 100644 index 29cb8a64fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/3e6aa765dd.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-22T11:02:38.689Z", - "Message": "Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\nMicrosoft.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-22T11:02:38.712Z", - "Message": "Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/6c84945046.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/6c84945046.json deleted file mode 100644 index 4819adc693..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/6c84945046.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f", - "Name": "Sales Automation", - "Path": "Sales Automation", - "Description": "Folder hosting the order search and reporting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e9f1c2d3-4a5b-46c7-8d9e-0f1a2b3c4d5e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/7a3dd5bdb6.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/7a3dd5bdb6.json deleted file mode 100644 index 01f22d89a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/7a3dd5bdb6.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c", - "StartTime": "2026-05-22T11:02:08.204Z", - "EndTime": "2026-05-22T11:02:38.731Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Info": "Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\n\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\r\n at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()\r\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, Dictionary`2 parameters, Int32 timeout)\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-05-22T11:02:07.990Z", - "ReleaseName": "OrderNotesSearch", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5042118, - "OrganizationUnitId": 7791624, - "FolderKey": "7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-22T11:02:38.731Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c", - "Id": 671209934 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/7a49de176b.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/7a49de176b.json deleted file mode 100644 index fbb92ed1de..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/7a49de176b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c", - "State": "Faulted", - "ProcessName": "OrderNotesSearch", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T11:02:08.204Z", - "EndTime": "2026-05-22T11:02:38.731Z", - "CreationTime": "2026-05-22T11:02:07.990Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/README.md deleted file mode 100644 index 8822a635ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# Execute Query Failure — Command Timeout Exceeded (Runaway Un-indexed Query) - -This scenario reproduces a runtime `Execute Query` failure caused by a -**query that exceeds the activity's `TimeoutMS`**. The activity runs an -expensive, un-indexed statement; the provider performs a full-table scan, -runs past the timeout, and aborts. Orchestrator surfaces -`Execute Query: Timeout expired. The timeout period elapsed prior to -completion of the operation or the server is not responding.` - -## What this scenario uncovers - -**Root Cause:** The `Execute Query` activity's `Sql` is -`SELECT * FROM Orders WHERE UPPER(Notes) LIKE '%...%'`. `UPPER(Notes)` -under a leading-wildcard `LIKE` is not sargable and `Orders` has no index -on `Notes`, so the provider does a full-table scan. `TimeoutMS` is left at -the default `30000` (30 s, **milliseconds**); the scan runs past 30 s and -the provider aborts with `Timeout expired`. The fix has two tiers: the -**real fix** is fixing the query/index (sargable predicate + supporting -index, narrower result set); raising `TimeoutMS` (in milliseconds) is only -a **stopgap** because it just delays the same failure on a runaway query. - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 5 — command timeout exceeded). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Execute Query with default `TimeoutMS` running an un-indexed full-scan query behind a valid Connect/Execute structure | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-5 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Sales Automation` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `OrderNotesSearch` job (ran ~30 s ≈ `TimeoutMS`). -3. `uip or jobs get --output json` → `Info` carries `Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.` -4. `uip or jobs logs --output json` → Trace lines showing the connection opened, the query started with `TimeoutMS=30000`, then `command aborted after 30000 ms ... full-table scan` — the smoking gun proving the query duration exceeded the timeout. -5. `process/Main.xaml` → the `Execute Query` `Sql` is the un-indexed `UPPER(Notes) LIKE '%...%'` scan with `TimeoutMS="30000"` (default). - -## How this differs from sibling branches - -| Dimension | branch 1 (null connection) | branch 3 (concatenation) | branch 5 (timeout) (this) | -|---|---|---|---| -| Connection opens? | no | yes | **yes** | -| SQL valid? | n/a | no (malformed) | **yes (just slow)** | -| Error signature | `NullReferenceException` | `A database error occurred` + `Incorrect syntax near ...` | **`Timeout expired ...`** | -| Where the fault is | wiring/scope | the SQL text | **query DURATION vs `TimeoutMS`** | -| Fix | wire/scope the connection | parameterize the query | **fix the index/query (real); raise `TimeoutMS` (stopgap)** | - -The failing layer here is the **query duration exceeding `TimeoutMS`**, -not the connection, the driver, the statement text, or a DB outage. The -agent must recognize this is a runaway un-indexed query — so the real fix -is the index/query, and raising `TimeoutMS` (in milliseconds) is only a -temporary stopgap. It must not blame a null connection, SQL syntax, a -provider mismatch, a CLR crash, the wrong activity, or "the database is -down". - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (Branch 5) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) name the `Timeout expired` failure at Execute Query, (b) attribute it to the query running longer than `TimeoutMS` because it is a runaway un-indexed full-table scan, and (c) recommend the real fix = add an index / narrow the query, noting raising `TimeoutMS` (milliseconds) is only a stopgap. -- Partial credit (0.8) if the agent matches the timeout branch but treats it as a legitimately long query and recommends only raising `TimeoutMS`. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-query-timeout-expired --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/manifest.json deleted file mode 100644 index 58784398da..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6c84945046.json" - }, - { - "match": "or jobs list --folder-key \"7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f\" --state Faulted --output json", - "file": "7a49de176b.json" - }, - { - "match": "or jobs list --folder-key 7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f --state Faulted --output json", - "file": "7a49de176b.json" - }, - { - "match": "or jobs get \"c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c\" --output json", - "file": "7a3dd5bdb6.json" - }, - { - "match": "or jobs get c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c --output json", - "file": "7a3dd5bdb6.json" - }, - { - "match": "or jobs logs \"c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c\" --level Error --output json", - "file": "3e6aa765dd.json" - }, - { - "match": "or jobs logs c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c --level Error --output json", - "file": "3e6aa765dd.json" - }, - { - "match": "or jobs logs \"c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c\" --output json", - "file": "357f120a5c.json" - }, - { - "match": "or jobs logs c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c --output json", - "file": "357f120a5c.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/uip-fixture.json new file mode 100644 index 0000000000..e01cd8aa83 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderNotesSearch\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T11:02:08.204Z\",\r\n \"EndTime\": \"2026-05-22T11:02:38.731Z\",\r\n \"CreationTime\": \"2026-05-22T11:02:07.990Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T11:02:38.689Z\",\r\n \"Message\": \"Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\\r\\nMicrosoft.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T11:02:38.712Z\",\r\n \"Message\": \"Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c\",\r\n \"StartTime\": \"2026-05-22T11:02:08.204Z\",\r\n \"EndTime\": \"2026-05-22T11:02:38.731Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> Microsoft.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\\r\\n at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()\\r\\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, Dictionary`2 parameters, Int32 timeout)\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-05-22T11:02:07.990Z\",\r\n \"ReleaseName\": \"OrderNotesSearch\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5042118,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-22T11:02:38.731Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c\",\r\n \"Id\": 671209934\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c4a7e2b9-5d1f-4e8a-9b3c-6d7e8f1a2b3c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-22T11:02:08.940Z\",\r\n \"Message\": \"OrderNotesSearch execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-22T11:02:09.011Z\",\r\n \"Message\": \"Connect to Database: connection opened to 'Server=sqlprod01;Database=Sales' using provider Microsoft.Data.SqlClient\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-22T11:02:09.114Z\",\r\n \"Message\": \"Execute Query: TimeoutMS=30000; executing SQL against connection 'Server=sqlprod01;Database=Sales': SELECT * FROM Orders WHERE UPPER(Notes) LIKE '%URGENT REFUND%'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-22T11:02:38.640Z\",\r\n \"Message\": \"Execute Query: command aborted after 30000 ms; the query did not complete within TimeoutMS. Orders has no index on Notes and UPPER(Notes) is not sargable, so the provider performed a full-table scan.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T11:02:38.689Z\",\r\n \"Message\": \"Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\\r\\nMicrosoft.Data.SqlClient.SqlException: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T11:02:38.712Z\",\r\n \"Message\": \"Execute Query: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7d2e9f1c-3a8b-4c5d-9e6f-2a1b3c4d5e6f\",\r\n \"Name\": \"Sales Automation\",\r\n \"Path\": \"Sales Automation\",\r\n \"Description\": \"Folder hosting the order search and reporting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e9f1c2d3-4a5b-46c7-8d9e-0f1a2b3c4d5e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/task.yaml index 3a99bd53ea..dda36f3e8a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-timeout-expired/task.yaml @@ -31,22 +31,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/README.md new file mode 100644 index 0000000000..69b5bec33d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/README.md @@ -0,0 +1,92 @@ +# Execute Query Failure — Wrong Activity for the Statement Type + +This scenario reproduces a runtime failure where an **`Execute Query`** +activity (which is for `SELECT` and returns a `DataTable`) is used to run +a **modification** statement — a `DELETE`. A modification produces no +result set, so `Execute Query` returns an **empty `DataTable`**. A +downstream `Assign` that reads `dt_Deleted.Rows(0)(0)` then throws +`System.ArgumentOutOfRangeException: There is no row at position 0`, +which Orchestrator surfaces as `Assign: There is no row at position 0.` + +The surface symptom is an `ArgumentOutOfRangeException` at the `Assign`. +The root cause is the **wrong activity choice** at the `Execute Query` — +the empty `DataTable` is the structural consequence of running a `DELETE` +through a `SELECT` activity. The differential is the whole point of this +scenario. + +## What this scenario uncovers + +**Root Cause:** `Execute Query` was used to run +`DELETE FROM TempStaging WHERE LoadDate < '2026-05-01'`. The correct +activity for a modification is `Execute Non Query`, whose `AffectedRecords` +output (an `Int32`) carries the affected-row count. The fix is to +**switch to `Execute Non Query`** and read `AffectedRecords` — **not** to +add an empty-`DataTable` guard / `Try-Catch` around the existing +`Execute Query` (the playbook's named anti-pattern, which hides the bug +and leaves the count unobtainable). + +This maps to: +`references/activity-packages/database-activities/playbooks/execute-query-failures.md` +(BRANCH 7 — wrong activity for the statement type). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — `targetFramework: "Windows"`, a clean `Connect to Database` (`Microsoft.Data.SqlClient`), an `Execute Query` running a `DELETE` into `dt_Deleted`, and an `Assign` reading `CInt(dt_Deleted.Rows(0)(0))` | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Data Operations` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `StagingCleanup` job. +3. `uip or jobs get --output json` → `Info` carries `Assign: There is no row at position 0.` wrapping `System.ArgumentOutOfRangeException: There is no row at position 0.` at `Assign "Compute deleted count"`. +4. `uip or jobs logs --output json` → the smoking-gun Trace lines: `Execute Query: executing 'DELETE FROM TempStaging WHERE LoadDate < ''2026-05-01''' (CommandType=Text); result set returned 0 rows`, then `Assign 'Compute deleted count': evaluating CInt(dt_Deleted.Rows(0)(0)) against an empty DataTable (0 rows)`. +5. `process/Main.xaml` → the `Execute Query` running a `DELETE` feeding an `Assign` that reads `Rows(0)`; `Connect to Database` is clean (`Microsoft.Data.SqlClient`). + +This Trace is what lets the agent connect "empty `DataTable`" → "Execute +Query was used for a `DELETE`" → Branch 7. Without it, the failure looks +like a generic downstream null/empty bug. + +## How this differs from sibling branches + +| Dimension | branch 1 (null connection) | branch 2 (provider mismatch) | branch 3 (concatenation) | branch 4 (query in conn string) | branch 5 (timeout) | branch 6 (CLR crash) | branch 7 (wrong activity) (this) | +|---|---|---|---|---|---|---|---| +| Connection opens? | no | no | yes | no | yes | yes | **yes** | +| Statement runs? | no | no | rejected | no | times out | crashes process | **yes — DELETE runs fine** | +| Error signature | `NullReferenceException` | `Keyword not supported` | `A database error occurred` + syntax | conn-string parse error | `Timeout expired` | exit `0xE0434352` | **`ArgumentOutOfRangeException: There is no row at position 0` at the Assign** | +| Where the fault is | wiring/scope | provider/driver | the SQL text | property misplacement | DB-side duration | managed runtime | **wrong activity choice; empty DataTable downstream** | +| Fix | wire/scope the connection | switch to Microsoft.Data.SqlClient | parameterize the query | move SQL to the Sql property | raise `TimeoutMS` / fix the query | bound the result set / update package | **switch to Execute Non Query, read AffectedRecords** | + +The failing layer here is the **activity choice**, not the connection, +the SQL text, a timeout, or the database. The `DELETE` ran successfully; +the empty `DataTable` is structural (a modification never returns rows), +so the `Rows(0)` fault is **not a data problem**. The agent must +recommend switching to `Execute Non Query` and reading `AffectedRecords` +— not adding an empty-`DataTable` guard / `Try-Catch`, and not blaming a +data problem, a null connection, a provider mismatch, a SQL syntax error, +a timeout, a CLR crash, or a DB outage. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the execute-query-failures playbook (branch 7) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) identify that `Execute Query` was used for a `DELETE` (wrong activity for a modification), (b) explain the empty `DataTable` → `Rows(0)` fault is the structural consequence, and (c) recommend switching to `Execute Non Query` and reading `AffectedRecords` (NOT adding an empty-row guard / Try-Catch). + +> Fixtures are **synthetic** — hand-authored from the documented branch-7 +> signature, not captured from a real tenant. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name execute-query-wrong-activity --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/17d594642f.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/17d594642f.json deleted file mode 100644 index 615a7f94d0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/17d594642f.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e", - "State": "Faulted", - "ProcessName": "StagingCleanup", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-29T03:05:00.402Z", - "EndTime": "2026-05-29T03:05:01.302Z", - "CreationTime": "2026-05-29T03:05:00.190Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/4156eeaa22.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/4156eeaa22.json deleted file mode 100644 index 963e1a4e68..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/4156eeaa22.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-29T03:05:00.640Z", - "Message": "StagingCleanup execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-29T03:05:00.918Z", - "Message": "Connect to Database: opened connection with ProviderName 'Microsoft.Data.SqlClient' to 'Server=sqlprod07;Database=Warehouse'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-29T03:05:01.187Z", - "Message": "Execute Query: executing 'DELETE FROM TempStaging WHERE LoadDate < ''2026-05-01''' (CommandType=Text); result set returned 0 rows" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-29T03:05:01.205Z", - "Message": "Assign 'Compute deleted count': evaluating CInt(dt_Deleted.Rows(0)(0)) against an empty DataTable (0 rows)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-29T03:05:01.241Z", - "Message": "Assign: There is no row at position 0.\r\nSystem.ArgumentOutOfRangeException: There is no row at position 0.\r\n at System.Data.DataRowCollection.get_Item(Int32 index)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-29T03:05:01.258Z", - "Message": "Assign: There is no row at position 0.\r\nin Main.xaml\r\n at Assign \"Compute deleted count\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/4aab646daf.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/4aab646daf.json deleted file mode 100644 index 58d3618c29..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/4aab646daf.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e", - "StartTime": "2026-05-29T03:05:00.402Z", - "EndTime": "2026-05-29T03:05:01.302Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "3c5a8d0f-2b4e-4a7c-9d1f-6e8a0b2c4d5f", - "Info": "Assign: There is no row at position 0.\r\n\r\nin Main.xaml\r\n at Assign \"Compute deleted count\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentOutOfRangeException: There is no row at position 0.\r\n at System.Data.DataRowCollection.get_Item(Int32 index)\r\n at Main.Compute deleted count_AssignExpression(ActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-05-29T03:05:00.190Z", - "ReleaseName": "StagingCleanup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5043207, - "OrganizationUnitId": 7791624, - "FolderKey": "8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-29T03:05:01.302Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e", - "Id": 671233907 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/README.md deleted file mode 100644 index f37d9450b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/README.md +++ /dev/null @@ -1,91 +0,0 @@ -# Execute Query Failure — Wrong Activity for the Statement Type - -This scenario reproduces a runtime failure where an **`Execute Query`** -activity (which is for `SELECT` and returns a `DataTable`) is used to run -a **modification** statement — a `DELETE`. A modification produces no -result set, so `Execute Query` returns an **empty `DataTable`**. A -downstream `Assign` that reads `dt_Deleted.Rows(0)(0)` then throws -`System.ArgumentOutOfRangeException: There is no row at position 0`, -which Orchestrator surfaces as `Assign: There is no row at position 0.` - -The surface symptom is an `ArgumentOutOfRangeException` at the `Assign`. -The root cause is the **wrong activity choice** at the `Execute Query` — -the empty `DataTable` is the structural consequence of running a `DELETE` -through a `SELECT` activity. The differential is the whole point of this -scenario. - -## What this scenario uncovers - -**Root Cause:** `Execute Query` was used to run -`DELETE FROM TempStaging WHERE LoadDate < '2026-05-01'`. The correct -activity for a modification is `Execute Non Query`, whose `AffectedRecords` -output (an `Int32`) carries the affected-row count. The fix is to -**switch to `Execute Non Query`** and read `AffectedRecords` — **not** to -add an empty-`DataTable` guard / `Try-Catch` around the existing -`Execute Query` (the playbook's named anti-pattern, which hides the bug -and leaves the count unobtainable). - -This maps to: -`references/activity-packages/database-activities/playbooks/execute-query-failures.md` -(BRANCH 7 — wrong activity for the statement type). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — `targetFramework: "Windows"`, a clean `Connect to Database` (`Microsoft.Data.SqlClient`), an `Execute Query` running a `DELETE` into `dt_Deleted`, and an `Assign` reading `CInt(dt_Deleted.Rows(0)(0))` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-7 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Data Operations` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `StagingCleanup` job. -3. `uip or jobs get --output json` → `Info` carries `Assign: There is no row at position 0.` wrapping `System.ArgumentOutOfRangeException: There is no row at position 0.` at `Assign "Compute deleted count"`. -4. `uip or jobs logs --output json` → the smoking-gun Trace lines: `Execute Query: executing 'DELETE FROM TempStaging WHERE LoadDate < ''2026-05-01''' (CommandType=Text); result set returned 0 rows`, then `Assign 'Compute deleted count': evaluating CInt(dt_Deleted.Rows(0)(0)) against an empty DataTable (0 rows)`. -5. `process/Main.xaml` → the `Execute Query` running a `DELETE` feeding an `Assign` that reads `Rows(0)`; `Connect to Database` is clean (`Microsoft.Data.SqlClient`). - -This Trace is what lets the agent connect "empty `DataTable`" → "Execute -Query was used for a `DELETE`" → Branch 7. Without it, the failure looks -like a generic downstream null/empty bug. - -## How this differs from sibling branches - -| Dimension | branch 1 (null connection) | branch 2 (provider mismatch) | branch 3 (concatenation) | branch 4 (query in conn string) | branch 5 (timeout) | branch 6 (CLR crash) | branch 7 (wrong activity) (this) | -|---|---|---|---|---|---|---|---| -| Connection opens? | no | no | yes | no | yes | yes | **yes** | -| Statement runs? | no | no | rejected | no | times out | crashes process | **yes — DELETE runs fine** | -| Error signature | `NullReferenceException` | `Keyword not supported` | `A database error occurred` + syntax | conn-string parse error | `Timeout expired` | exit `0xE0434352` | **`ArgumentOutOfRangeException: There is no row at position 0` at the Assign** | -| Where the fault is | wiring/scope | provider/driver | the SQL text | property misplacement | DB-side duration | managed runtime | **wrong activity choice; empty DataTable downstream** | -| Fix | wire/scope the connection | switch to Microsoft.Data.SqlClient | parameterize the query | move SQL to the Sql property | raise `TimeoutMS` / fix the query | bound the result set / update package | **switch to Execute Non Query, read AffectedRecords** | - -The failing layer here is the **activity choice**, not the connection, -the SQL text, a timeout, or the database. The `DELETE` ran successfully; -the empty `DataTable` is structural (a modification never returns rows), -so the `Rows(0)` fault is **not a data problem**. The agent must -recommend switching to `Execute Non Query` and reading `AffectedRecords` -— not adding an empty-`DataTable` guard / `Try-Catch`, and not blaming a -data problem, a null connection, a provider mismatch, a SQL syntax error, -a timeout, a CLR crash, or a DB outage. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the execute-query-failures playbook (branch 7) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) identify that `Execute Query` was used for a `DELETE` (wrong activity for a modification), (b) explain the empty `DataTable` → `Rows(0)` fault is the structural consequence, and (c) recommend switching to `Execute Non Query` and reading `AffectedRecords` (NOT adding an empty-row guard / Try-Catch). - -> Fixtures are **synthetic** — hand-authored from the documented branch-7 -> signature, not captured from a real tenant. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name execute-query-wrong-activity --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/a3e11dadda.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/a3e11dadda.json deleted file mode 100644 index bba2dccf12..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/a3e11dadda.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-29T03:05:01.241Z", - "Message": "Assign: There is no row at position 0.\r\nSystem.ArgumentOutOfRangeException: There is no row at position 0.\r\n at System.Data.DataRowCollection.get_Item(Int32 index)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-29T03:05:01.258Z", - "Message": "Assign: There is no row at position 0.\r\nin Main.xaml\r\n at Assign \"Compute deleted count\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/aad6da9cf6.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/aad6da9cf6.json deleted file mode 100644 index b901cf395f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/aad6da9cf6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d", - "Name": "Data Operations", - "Path": "Data Operations", - "Description": "Folder hosting the nightly warehouse maintenance and staging-cleanup processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b1d6f0a2-7c3e-4d59-8a1b-4f0c2e6d9a37", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/manifest.json deleted file mode 100644 index a87a69c0a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Execute Query scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "aad6da9cf6.json" - }, - { - "match": "or jobs list --folder-key \"8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d\" --state Faulted --output json", - "file": "17d594642f.json" - }, - { - "match": "or jobs list --folder-key 8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d --state Faulted --output json", - "file": "17d594642f.json" - }, - { - "match": "or jobs get \"9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e\" --output json", - "file": "4aab646daf.json" - }, - { - "match": "or jobs get 9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e --output json", - "file": "4aab646daf.json" - }, - { - "match": "or jobs logs \"9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e\" --level Error --output json", - "file": "a3e11dadda.json" - }, - { - "match": "or jobs logs 9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e --level Error --output json", - "file": "a3e11dadda.json" - }, - { - "match": "or jobs logs \"9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e\" --output json", - "file": "4156eeaa22.json" - }, - { - "match": "or jobs logs 9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e --output json", - "file": "4156eeaa22.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/uip-fixture.json new file mode 100644 index 0000000000..b08933fc19 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"StagingCleanup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-29T03:05:00.402Z\",\r\n \"EndTime\": \"2026-05-29T03:05:01.302Z\",\r\n \"CreationTime\": \"2026-05-29T03:05:00.190Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T03:05:01.241Z\",\r\n \"Message\": \"Assign: There is no row at position 0.\\r\\nSystem.ArgumentOutOfRangeException: There is no row at position 0.\\r\\n at System.Data.DataRowCollection.get_Item(Int32 index)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T03:05:01.258Z\",\r\n \"Message\": \"Assign: There is no row at position 0.\\r\\nin Main.xaml\\r\\n at Assign \\\"Compute deleted count\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e\",\r\n \"StartTime\": \"2026-05-29T03:05:00.402Z\",\r\n \"EndTime\": \"2026-05-29T03:05:01.302Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"3c5a8d0f-2b4e-4a7c-9d1f-6e8a0b2c4d5f\",\r\n \"Info\": \"Assign: There is no row at position 0.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Compute deleted count\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentOutOfRangeException: There is no row at position 0.\\r\\n at System.Data.DataRowCollection.get_Item(Int32 index)\\r\\n at Main.Compute deleted count_AssignExpression(ActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-05-29T03:05:00.190Z\",\r\n \"ReleaseName\": \"StagingCleanup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5043207,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-29T03:05:01.302Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e\",\r\n \"Id\": 671233907\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9d3f7b2e-5a8c-4f1d-b6e9-2c4a8d0f3b5e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-29T03:05:00.640Z\",\r\n \"Message\": \"StagingCleanup execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-29T03:05:00.918Z\",\r\n \"Message\": \"Connect to Database: opened connection with ProviderName 'Microsoft.Data.SqlClient' to 'Server=sqlprod07;Database=Warehouse'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-29T03:05:01.187Z\",\r\n \"Message\": \"Execute Query: executing 'DELETE FROM TempStaging WHERE LoadDate < ''2026-05-01''' (CommandType=Text); result set returned 0 rows\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-29T03:05:01.205Z\",\r\n \"Message\": \"Assign 'Compute deleted count': evaluating CInt(dt_Deleted.Rows(0)(0)) against an empty DataTable (0 rows)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T03:05:01.241Z\",\r\n \"Message\": \"Assign: There is no row at position 0.\\r\\nSystem.ArgumentOutOfRangeException: There is no row at position 0.\\r\\n at System.Data.DataRowCollection.get_Item(Int32 index)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-29T03:05:01.258Z\",\r\n \"Message\": \"Assign: There is no row at position 0.\\r\\nin Main.xaml\\r\\n at Assign \\\"Compute deleted count\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8c4e1a7d-3f6b-4e2a-9d8c-0a5f3b7e1c4d\",\r\n \"Name\": \"Data Operations\",\r\n \"Path\": \"Data Operations\",\r\n \"Description\": \"Folder hosting the nightly warehouse maintenance and staging-cleanup processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b1d6f0a2-7c3e-4d59-8a1b-4f0c2e6d9a37\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/task.yaml index b89c20d062..b7798942d1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-execute-query-wrong-activity/task.yaml @@ -32,22 +32,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/README.md new file mode 100644 index 0000000000..e784c9fbbf --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/README.md @@ -0,0 +1,102 @@ +# Start Transaction Failure — Scope Skips Its Body (Package v1.5.0 Bug) + +This scenario reproduces a `Start Transaction` failure where the **job +reports Success** every night, yet the reconciliation it should perform +never happens — **no rows are written and there is no error**. The project +pins `UiPath.Database.Activities` **v1.5.0**, a defective build in which the +`Start Transaction` scope **skips its body entirely**: the scope enters and +exits cleanly, committing an empty transaction, and **none** of its child +activities ever execute. + +## What this scenario uncovers + +**Root Cause:** The `UiPath.Database.Activities` v1.5.0 build has a defect +where the `Start Transaction` (`DatabaseTransaction`) scope never invokes +its body. The well-formed child activities inside the scope — a +parameterized `Execute Non Query` (the reconciliation `UPDATE`, bound to the +scope's `dbTxn` via `ExistingDbConnection`) and an `Info` `Log Message` — +are simply skipped. Nothing throws, nothing rolls back, and the scope +commits an empty transaction, so Orchestrator records a clean Success on a +job that accomplished nothing. The fix is a **package-version change**: +upgrade `UiPath.Database.Activities` to **v1.7.1+** (or downgrade to a +known-good **v1.4.0**) via Manage Packages — it is **not** a workflow +change. + +This maps to: +`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` +(BRANCH 2 — transaction scope skips its body). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — a `Start Transaction` (`ui:DatabaseTransaction`, `Microsoft.Data.SqlClient`) scope wrapping a `Sequence` with a parameterized `Execute Non Query` (`ExistingDbConnection="[dbTxn]"`) and a `Log Message` (Info); `project.json` pins `UiPath.Database.Activities` **`[1.5.0]`** | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Finance Operations` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → **empty** list (no faulted job; the failure hides behind a green Success). +3. `uip or jobs list --folder-key ` (no state) / `--state Successful` → the single **Successful** `NightlyReconcileTxn` job. +4. `uip or jobs get --output json` → `State: "Successful"`, benign `Info: "Execution ended."`. +5. `uip or jobs logs --level Error --output json` → **empty** list — there is no error at any level. +6. `uip or jobs logs --output json` → the smoking gun: the `Start Transaction` scope starts on `UiPath.Database.Activities v1.5.0` and **immediately** commits/exits, with **zero** child-activity log lines in between and no error anywhere. +7. `process/project.json` → `"UiPath.Database.Activities": "[1.5.0]"` — the defective build. +8. `process/Main.xaml` → the child `Execute Non Query` + `Log Message` that should have run but never executed. + +## How this differs from sibling branches + +| Dimension | branch 1 (silent rollback) | branch 2 (package body-skip) (this) | branch 3 (provider mismatch) | branch 4 (connection not propagated) | +|---|---|---|---|---| +| Job state | Successful (false green) | **Successful (false green)** | Faulted | Faulted (or non-transactional writes) | +| Child activities run? | yes — and one errored | **no — body skipped entirely** | n/a (fails at connect) | yes | +| Error present? | yes — a child Error, then swallowed | **no error anywhere** | yes — type-initializer crash | yes — null-ref / instant timeout | +| Signature | child Error + "caught, continuing" Info, then commit | **empty scope (start then immediate commit), no child logs, no error, on v1.5.0** | `Keyword not supported` / type-initializer crash | `Object reference not set` / instant `Timeout expired`, or auto-committed writes | +| Where the fault is | swallowed in-scope catch (no Throw) | **package version (v1.5.0 defect)** | provider/driver after migration | child `ExistingDbConnection` wiring | +| Fix | re-raise (Throw) so the transaction rolls back | **change the package version (upgrade to v1.7.1+ / downgrade to v1.4.0)** | switch to `Microsoft.Data.SqlClient` | thread the scope's `DatabaseConnection` into every child | + +**The branch-1-vs-branch-2 distinction is the crux of this test.** Both +report a false-green Success, but they are opposites in the logs: + +- **Branch 1 (silent rollback):** a child activity **ran and errored**, and + the error was **swallowed** by a no-`Throw` catch. The logs carry a child + Error line immediately followed by a "caught and continuing" line. +- **Branch 2 (this scenario):** **no child ran at all** and there is **no + error anywhere**. The logs show only the scope start and the scope commit + with nothing in between. The tell is the empty scope plus the pinned + v1.5.0 package version — there is no swallowed exception to find. + +The failing layer here is the **`UiPath.Database.Activities` package +version**, not the workflow logic, the exception flow, the connection +wiring, the provider, the data, or the schedule. The agent must recommend a +**version change** (upgrade to v1.7.1+ or downgrade to v1.4.0) — not a +Try Catch, a Throw, a retry, added logging, or any workflow edit. + +## Synthetic fixtures + +documented branch-2 signature, not captured from a real +`.local/investigations/` run. They are internally consistent (one +Successful job, an empty Faulted list, an empty error-level log, and job +the full discovery chain. Replace them with verbatim captures from a real +investigation before treating this scenario as a hard regression signal. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the start-transaction-failures playbook (branch 2) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) identify that the `Start Transaction` scope executed none of its child activities with no error, (b) attribute it to the defective `UiPath.Database.Activities` v1.5.0 build (body-skip bug), and (c) recommend upgrading to v1.7.1+ (or downgrading to v1.4.0). + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name start-transaction-body-skip --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/0ccdf3ced8.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/0ccdf3ced8.json deleted file mode 100644 index f13816722e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/0ccdf3ced8.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d", - "StartTime": "2026-05-31T01:30:01.118Z", - "EndTime": "2026-05-31T01:30:03.402Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "e6c9d4f0-3a5b-4d7c-8e1f-2a3b4c5d6e7f", - "Info": "Execution ended.", - "CreationTime": "2026-05-31T01:30:00.904Z", - "ReleaseName": "NightlyReconcileTxn", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5048217, - "OrganizationUnitId": 7791624, - "FolderKey": "c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-31T01:30:03.402Z", - "JobError": {}, - "ErrorCode": "", - "TraceId": "d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d", - "Id": 671320515 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/5706c9815b.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/5706c9815b.json deleted file mode 100644 index 916de24269..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/5706c9815b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d", - "State": "Successful", - "ProcessName": "NightlyReconcileTxn", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-31T01:30:01.118Z", - "EndTime": "2026-05-31T01:30:03.402Z", - "CreationTime": "2026-05-31T01:30:00.904Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/README.md deleted file mode 100644 index dc9856d2c6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/README.md +++ /dev/null @@ -1,103 +0,0 @@ -# Start Transaction Failure — Scope Skips Its Body (Package v1.5.0 Bug) - -This scenario reproduces a `Start Transaction` failure where the **job -reports Success** every night, yet the reconciliation it should perform -never happens — **no rows are written and there is no error**. The project -pins `UiPath.Database.Activities` **v1.5.0**, a defective build in which the -`Start Transaction` scope **skips its body entirely**: the scope enters and -exits cleanly, committing an empty transaction, and **none** of its child -activities ever execute. - -## What this scenario uncovers - -**Root Cause:** The `UiPath.Database.Activities` v1.5.0 build has a defect -where the `Start Transaction` (`DatabaseTransaction`) scope never invokes -its body. The well-formed child activities inside the scope — a -parameterized `Execute Non Query` (the reconciliation `UPDATE`, bound to the -scope's `dbTxn` via `ExistingDbConnection`) and an `Info` `Log Message` — -are simply skipped. Nothing throws, nothing rolls back, and the scope -commits an empty transaction, so Orchestrator records a clean Success on a -job that accomplished nothing. The fix is a **package-version change**: -upgrade `UiPath.Database.Activities` to **v1.7.1+** (or downgrade to a -known-good **v1.4.0**) via Manage Packages — it is **not** a workflow -change. - -This maps to: -`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` -(BRANCH 2 — transaction scope skips its body). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — a `Start Transaction` (`ui:DatabaseTransaction`, `Microsoft.Data.SqlClient`) scope wrapping a `Sequence` with a parameterized `Execute Non Query` (`ExistingDbConnection="[dbTxn]"`) and a `Log Message` (Info); `project.json` pins `UiPath.Database.Activities` **`[1.5.0]`** | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Finance Operations` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → **empty** list (no faulted job; the failure hides behind a green Success). -3. `uip or jobs list --folder-key ` (no state) / `--state Successful` → the single **Successful** `NightlyReconcileTxn` job. -4. `uip or jobs get --output json` → `State: "Successful"`, benign `Info: "Execution ended."`. -5. `uip or jobs logs --level Error --output json` → **empty** list — there is no error at any level. -6. `uip or jobs logs --output json` → the smoking gun: the `Start Transaction` scope starts on `UiPath.Database.Activities v1.5.0` and **immediately** commits/exits, with **zero** child-activity log lines in between and no error anywhere. -7. `process/project.json` → `"UiPath.Database.Activities": "[1.5.0]"` — the defective build. -8. `process/Main.xaml` → the child `Execute Non Query` + `Log Message` that should have run but never executed. - -## How this differs from sibling branches - -| Dimension | branch 1 (silent rollback) | branch 2 (package body-skip) (this) | branch 3 (provider mismatch) | branch 4 (connection not propagated) | -|---|---|---|---|---| -| Job state | Successful (false green) | **Successful (false green)** | Faulted | Faulted (or non-transactional writes) | -| Child activities run? | yes — and one errored | **no — body skipped entirely** | n/a (fails at connect) | yes | -| Error present? | yes — a child Error, then swallowed | **no error anywhere** | yes — type-initializer crash | yes — null-ref / instant timeout | -| Signature | child Error + "caught, continuing" Info, then commit | **empty scope (start then immediate commit), no child logs, no error, on v1.5.0** | `Keyword not supported` / type-initializer crash | `Object reference not set` / instant `Timeout expired`, or auto-committed writes | -| Where the fault is | swallowed in-scope catch (no Throw) | **package version (v1.5.0 defect)** | provider/driver after migration | child `ExistingDbConnection` wiring | -| Fix | re-raise (Throw) so the transaction rolls back | **change the package version (upgrade to v1.7.1+ / downgrade to v1.4.0)** | switch to `Microsoft.Data.SqlClient` | thread the scope's `DatabaseConnection` into every child | - -**The branch-1-vs-branch-2 distinction is the crux of this test.** Both -report a false-green Success, but they are opposites in the logs: - -- **Branch 1 (silent rollback):** a child activity **ran and errored**, and - the error was **swallowed** by a no-`Throw` catch. The logs carry a child - Error line immediately followed by a "caught and continuing" line. -- **Branch 2 (this scenario):** **no child ran at all** and there is **no - error anywhere**. The logs show only the scope start and the scope commit - with nothing in between. The tell is the empty scope plus the pinned - v1.5.0 package version — there is no swallowed exception to find. - -The failing layer here is the **`UiPath.Database.Activities` package -version**, not the workflow logic, the exception flow, the connection -wiring, the provider, the data, or the schedule. The agent must recommend a -**version change** (upgrade to v1.7.1+ or downgrade to v1.4.0) — not a -Try Catch, a Throw, a retry, added logging, or any workflow edit. - -## Synthetic fixtures - -The `fixtures/mocks/responses/*.json` files are **hand-authored** from the -documented branch-2 signature, not captured from a real -`.local/investigations/` run. They are internally consistent (one -Successful job, an empty Faulted list, an empty error-level log, and job -logs carrying the empty-scope smoking gun) so the mock dispatcher resolves -the full discovery chain. Replace them with verbatim captures from a real -investigation before treating this scenario as a hard regression signal. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the start-transaction-failures playbook (branch 2) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) identify that the `Start Transaction` scope executed none of its child activities with no error, (b) attribute it to the defective `UiPath.Database.Activities` v1.5.0 build (body-skip bug), and (c) recommend upgrading to v1.7.1+ (or downgrading to v1.4.0). - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name start-transaction-body-skip --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/a758c4940e.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/a758c4940e.json deleted file mode 100644 index 5059ea6d95..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/a758c4940e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-31T01:30:01.402Z", - "Message": "NightlyReconcileTxn execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-31T01:30:02.118Z", - "Message": "Start Transaction: scope started (UiPath.Database.Activities v1.5.0)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-31T01:30:02.305Z", - "Message": "Start Transaction: scope exited; committing transaction" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-31T01:30:03.291Z", - "Message": "NightlyReconcileTxn execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/b6157da8ba.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/b6157da8ba.json deleted file mode 100644 index 32b20f5cb0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/b6157da8ba.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c", - "Name": "Finance Operations", - "Path": "Finance Operations", - "Description": "Folder hosting the ledger posting and reconciliation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a7c9e1b3-5d7f-4a2c-8b6d-9e0f1a2b3c4d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/manifest.json deleted file mode 100644 index 3123a8ee3f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Start Transaction scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "b6157da8ba.json" - }, - { - "match": "or jobs list --folder-key \"c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c\" --state Successful --output json", - "file": "5706c9815b.json" - }, - { - "match": "or jobs list --folder-key c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c --state Successful --output json", - "file": "5706c9815b.json" - }, - { - "match": "or jobs list --folder-key \"c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c\" --output json", - "file": "5706c9815b.json" - }, - { - "match": "or jobs list --folder-key c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c --output json", - "file": "5706c9815b.json" - }, - { - "match": "or jobs get \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\" --output json", - "file": "0ccdf3ced8.json" - }, - { - "match": "or jobs get d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d --output json", - "file": "0ccdf3ced8.json" - }, - { - "match": "or jobs logs \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\" --output json", - "file": "a758c4940e.json" - }, - { - "match": "or jobs logs d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d --output json", - "file": "a758c4940e.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/uip-fixture.json new file mode 100644 index 0000000000..87326780ba --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NightlyReconcileTxn\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-31T01:30:01.118Z\",\r\n \"EndTime\": \"2026-05-31T01:30:03.402Z\",\r\n \"CreationTime\": \"2026-05-31T01:30:00.904Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NightlyReconcileTxn\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-31T01:30:01.118Z\",\r\n \"EndTime\": \"2026-05-31T01:30:03.402Z\",\r\n \"CreationTime\": \"2026-05-31T01:30:00.904Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\",\r\n \"StartTime\": \"2026-05-31T01:30:01.118Z\",\r\n \"EndTime\": \"2026-05-31T01:30:03.402Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"e6c9d4f0-3a5b-4d7c-8e1f-2a3b4c5d6e7f\",\r\n \"Info\": \"Execution ended.\",\r\n \"CreationTime\": \"2026-05-31T01:30:00.904Z\",\r\n \"ReleaseName\": \"NightlyReconcileTxn\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5048217,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-31T01:30:03.402Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d\",\r\n \"Id\": 671320515\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d5b8f3a0-2c4e-4f6b-9d7a-0e1f2a3b4c5d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-31T01:30:01.402Z\",\r\n \"Message\": \"NightlyReconcileTxn execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-31T01:30:02.118Z\",\r\n \"Message\": \"Start Transaction: scope started (UiPath.Database.Activities v1.5.0)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-31T01:30:02.305Z\",\r\n \"Message\": \"Start Transaction: scope exited; committing transaction\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-31T01:30:03.291Z\",\r\n \"Message\": \"NightlyReconcileTxn execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c4a7e2f9-1b3d-4e5a-8c6f-9d0e1f2a3b4c\",\r\n \"Name\": \"Finance Operations\",\r\n \"Path\": \"Finance Operations\",\r\n \"Description\": \"Folder hosting the ledger posting and reconciliation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a7c9e1b3-5d7f-4a2c-8b6d-9e0f1a2b3c4d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/task.yaml index af1bb8941f..a7751ddcc3 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-body-skip/task.yaml @@ -32,22 +32,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/README.md new file mode 100644 index 0000000000..46da876541 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/README.md @@ -0,0 +1,79 @@ +# Start Transaction Failure — Connection Not Propagated to Child + +This scenario reproduces a runtime `Start Transaction` +(`DatabaseTransaction`) failure where the scope opens its connection +**successfully**, but a child `Execute Query` inside the scope body is +left with an **empty `ExistingDbConnection`** — it is never bound to the +scope's output `DatabaseConnection` (`dbTxn`). At runtime the child runs +with a `null` connection and throws +`System.NullReferenceException: Object reference not set to an instance of +an object`, which Orchestrator surfaces as +`Execute Query: Object reference not set to an instance of an object`. + +## What this scenario uncovers + +**Root Cause:** The `Start Transaction` scope connected fine on +`Microsoft.Data.SqlClient` and assigned a live `DatabaseConnection` to +`dbTxn`. The child `Execute Query` inside the scope, however, has no +`ExistingDbConnection` binding (the property is empty, not `[dbTxn]`), so +it executes against a null connection. The transaction's connection was +**never propagated** to the child. The fix is to **bind `dbTxn` into the +child's `ExistingDbConnection`** so the child runs inside the +transaction — not to wrap it in a Try Catch, add a null check, or give +the child its own inline connection. + +This maps to: +`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` +(BRANCH 4 — output connection not propagated to child activities). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — a `Start Transaction` scope (`Microsoft.Data.SqlClient`, output `dbTxn`) wrapping an `Execute Query` whose `ExistingDbConnection` is left empty (not bound to `dbTxn`) | + +The decisive evidence chain: + +1. `uip or folders list --output json` -> the `Sales Operations` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` -> the single faulted `OrderSettlementTxn` job. +3. `uip or jobs get --output json` -> `Info` carries `Execute Query: Object reference not set to an instance of an object.` wrapping `System.NullReferenceException`, with the stack at `ExecuteQuery "Execute Query"` -> `DatabaseTransaction "Start Transaction"` -> `Sequence "Main Sequence"` -> `Main "Main"`. +4. `uip or jobs logs --output json` -> the smoking gun: a Trace showing `Start Transaction` opened its connection and assigned it to `dbTxn` **successfully**, then a Trace showing `Execute Query` ran with `ExistingDbConnection = (null)`, then the `NullReferenceException` Error lines. +5. `process/Main.xaml` -> the child `Execute Query` has no `ExistingDbConnection` binding while the scope outputs `DatabaseConnection` to `dbTxn`; `process/project.json` -> `UiPath.Database.Activities [1.7.0]` (a healthy build, ruling out branch 2). + +## How this differs from sibling branches + +| Dimension | branch 1 (silent rollback) | branch 2 (body skip) | branch 3 (provider/type after migration) | branch 4 (connection not propagated) (this) | +|---|---|---|---|---| +| Job state | Success | Success | Faulted | **Faulted** | +| Scope connection opens? | yes | n/a | **no — type/provider crash** | **yes — opens fine** | +| Child runs? | yes (errored, swallowed) | **no — body skipped** | no | **yes — but with null connection** | +| Error signature | none surfaced | none | type-initializer / unresolvable type | **`NullReferenceException` at the child** | +| Fix | re-throw in catch so scope rolls back | move off the defective package version | switch to `Microsoft.Data.SqlClient` + clear cache | **bind `dbTxn` into the child's `ExistingDbConnection`** | + +The failing layer here is the **child's connection wiring**, not the +scope's connection (which opened fine), not a swallowed rollback, not a +package-version body-skip, not a post-migration provider crash, not a bad +connection string, not SQL syntax, and not a timeout. The agent must +recommend **propagating the scope's connection** (`dbTxn`) into the +child's `ExistingDbConnection` — not blame the connection string, a +provider mismatch, a package bug, a swallowed rollback, or a timeout, and +not settle for a Try Catch / null check as the primary fix. + +## Synthetic fixtures + +documented branch-4 signature, not captured from a live tenant. All +identifiers (folder key, job key, machine, account) are synthetic and +valid hex. Replace with verbatim captures from a real +`.local/investigations/` before treating this as a hard regression signal. + +## Success criteria + +The test scores the **conclusion, not the trajectory**: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the start-transaction-failures playbook (branch 4) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) identify that the child `Execute Query`'s `ExistingDbConnection` is empty / not bound to the `Start Transaction` output, (b) explain that is why it threw `NullReferenceException` while the scope's own connection was fine, and (c) recommend binding `dbTxn` into the child's `ExistingDbConnection`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/2befd312ea.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/2befd312ea.json deleted file mode 100644 index 80c719d39b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/2befd312ea.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-30T08:35:00.640Z", - "Message": "OrderSettlementTxn execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-30T08:35:01.502Z", - "Message": "Start Transaction: opened connection with ProviderName 'Microsoft.Data.SqlClient' and connection string 'Server=sqlsales02;Database=Orders;Integrated Security=true;Encrypt=true'; transaction begun, connection assigned to variable 'dbTxn'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-30T08:35:01.731Z", - "Message": "Execute Query: executing with ExistingDbConnection = (null) [property is empty - not bound to 'dbTxn']; Sql = 'SELECT OrderId, Status FROM Orders WHERE Status = 'Pending''" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T08:35:01.788Z", - "Message": "Execute Query: Object reference not set to an instance of an object.\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, CommandType commandType, Dictionary`2 parameters, Int32 timeout)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T08:35:01.804Z", - "Message": "Execute Query: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at DatabaseTransaction \"Start Transaction\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/2d07d5456e.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/2d07d5456e.json deleted file mode 100644 index afb019d03b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/2d07d5456e.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234", - "StartTime": "2026-05-30T08:35:00.331Z", - "EndTime": "2026-05-30T08:35:01.842Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Info": "Execute Query: Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at DatabaseTransaction \"Start Transaction\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, CommandType commandType, Dictionary`2 parameters, Int32 timeout)\r\n at UiPath.Database.Activities.ExecuteQuery.ExecuteInternal(AsyncCodeActivityContext context)\r\n at UiPath.Database.Activities.ExecuteQuery.BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, Object state)", - "CreationTime": "2026-05-30T08:35:00.118Z", - "ReleaseName": "OrderSettlementTxn", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5042118, - "OrganizationUnitId": 7791624, - "FolderKey": "f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T08:35:01.842Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234", - "Id": 671208842 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/4cce4b34b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/4cce4b34b6.json deleted file mode 100644 index f97c4be1c1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/4cce4b34b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012", - "Name": "Sales Operations", - "Path": "Sales Operations", - "Description": "Folder hosting the order settlement and reconciliation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c4f1a983-6b2d-4e57-9a1c-3d8e0f5b7a26", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/97fa0fbf36.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/97fa0fbf36.json deleted file mode 100644 index a8022eaac2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/97fa0fbf36.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-30T08:35:01.788Z", - "Message": "Execute Query: Object reference not set to an instance of an object.\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, CommandType commandType, Dictionary`2 parameters, Int32 timeout)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T08:35:01.804Z", - "Message": "Execute Query: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at ExecuteQuery \"Execute Query\"\r\n at DatabaseTransaction \"Start Transaction\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/9c3ac04d7c.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/9c3ac04d7c.json deleted file mode 100644 index 28e99e961e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/9c3ac04d7c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234", - "State": "Faulted", - "ProcessName": "OrderSettlementTxn", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-30T08:35:00.331Z", - "EndTime": "2026-05-30T08:35:01.842Z", - "CreationTime": "2026-05-30T08:35:00.118Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/README.md deleted file mode 100644 index 0ab839b438..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/README.md +++ /dev/null @@ -1,79 +0,0 @@ -# Start Transaction Failure — Connection Not Propagated to Child - -This scenario reproduces a runtime `Start Transaction` -(`DatabaseTransaction`) failure where the scope opens its connection -**successfully**, but a child `Execute Query` inside the scope body is -left with an **empty `ExistingDbConnection`** — it is never bound to the -scope's output `DatabaseConnection` (`dbTxn`). At runtime the child runs -with a `null` connection and throws -`System.NullReferenceException: Object reference not set to an instance of -an object`, which Orchestrator surfaces as -`Execute Query: Object reference not set to an instance of an object`. - -## What this scenario uncovers - -**Root Cause:** The `Start Transaction` scope connected fine on -`Microsoft.Data.SqlClient` and assigned a live `DatabaseConnection` to -`dbTxn`. The child `Execute Query` inside the scope, however, has no -`ExistingDbConnection` binding (the property is empty, not `[dbTxn]`), so -it executes against a null connection. The transaction's connection was -**never propagated** to the child. The fix is to **bind `dbTxn` into the -child's `ExistingDbConnection`** so the child runs inside the -transaction — not to wrap it in a Try Catch, add a null check, or give -the child its own inline connection. - -This maps to: -`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` -(BRANCH 4 — output connection not propagated to child activities). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — a `Start Transaction` scope (`Microsoft.Data.SqlClient`, output `dbTxn`) wrapping an `Execute Query` whose `ExistingDbConnection` is left empty (not bound to `dbTxn`) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-4 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` -> the `Sales Operations` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` -> the single faulted `OrderSettlementTxn` job. -3. `uip or jobs get --output json` -> `Info` carries `Execute Query: Object reference not set to an instance of an object.` wrapping `System.NullReferenceException`, with the stack at `ExecuteQuery "Execute Query"` -> `DatabaseTransaction "Start Transaction"` -> `Sequence "Main Sequence"` -> `Main "Main"`. -4. `uip or jobs logs --output json` -> the smoking gun: a Trace showing `Start Transaction` opened its connection and assigned it to `dbTxn` **successfully**, then a Trace showing `Execute Query` ran with `ExistingDbConnection = (null)`, then the `NullReferenceException` Error lines. -5. `process/Main.xaml` -> the child `Execute Query` has no `ExistingDbConnection` binding while the scope outputs `DatabaseConnection` to `dbTxn`; `process/project.json` -> `UiPath.Database.Activities [1.7.0]` (a healthy build, ruling out branch 2). - -## How this differs from sibling branches - -| Dimension | branch 1 (silent rollback) | branch 2 (body skip) | branch 3 (provider/type after migration) | branch 4 (connection not propagated) (this) | -|---|---|---|---|---| -| Job state | Success | Success | Faulted | **Faulted** | -| Scope connection opens? | yes | n/a | **no — type/provider crash** | **yes — opens fine** | -| Child runs? | yes (errored, swallowed) | **no — body skipped** | no | **yes — but with null connection** | -| Error signature | none surfaced | none | type-initializer / unresolvable type | **`NullReferenceException` at the child** | -| Fix | re-throw in catch so scope rolls back | move off the defective package version | switch to `Microsoft.Data.SqlClient` + clear cache | **bind `dbTxn` into the child's `ExistingDbConnection`** | - -The failing layer here is the **child's connection wiring**, not the -scope's connection (which opened fine), not a swallowed rollback, not a -package-version body-skip, not a post-migration provider crash, not a bad -connection string, not SQL syntax, and not a timeout. The agent must -recommend **propagating the scope's connection** (`dbTxn`) into the -child's `ExistingDbConnection` — not blame the connection string, a -provider mismatch, a package bug, a swallowed rollback, or a timeout, and -not settle for a Try Catch / null check as the primary fix. - -## Synthetic fixtures - -The `fixtures/mocks/responses/*.json` files are **hand-authored** from the -documented branch-4 signature, not captured from a live tenant. All -identifiers (folder key, job key, machine, account) are synthetic and -valid hex. Replace with verbatim captures from a real -`.local/investigations/` before treating this as a hard regression signal. - -## Success criteria - -The test scores the **conclusion, not the trajectory**: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the start-transaction-failures playbook (branch 4) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) identify that the child `Execute Query`'s `ExistingDbConnection` is empty / not bound to the `Start Transaction` output, (b) explain that is why it threw `NullReferenceException` while the scope's own connection was fine, and (c) recommend binding `dbTxn` into the child's `ExistingDbConnection`. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/manifest.json deleted file mode 100644 index 049867ace9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Start Transaction scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "4cce4b34b6.json" - }, - { - "match": "or jobs list --folder-key \"f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012\" --state Faulted --output json", - "file": "9c3ac04d7c.json" - }, - { - "match": "or jobs list --folder-key f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012 --state Faulted --output json", - "file": "9c3ac04d7c.json" - }, - { - "match": "or jobs get \"a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234\" --output json", - "file": "2d07d5456e.json" - }, - { - "match": "or jobs get a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234 --output json", - "file": "2d07d5456e.json" - }, - { - "match": "or jobs logs \"a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234\" --level Error --output json", - "file": "97fa0fbf36.json" - }, - { - "match": "or jobs logs a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234 --level Error --output json", - "file": "97fa0fbf36.json" - }, - { - "match": "or jobs logs \"a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234\" --output json", - "file": "2befd312ea.json" - }, - { - "match": "or jobs logs a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234 --output json", - "file": "2befd312ea.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/uip-fixture.json new file mode 100644 index 0000000000..1e5ec6d1e0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderSettlementTxn\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-30T08:35:00.331Z\",\r\n \"EndTime\": \"2026-05-30T08:35:01.842Z\",\r\n \"CreationTime\": \"2026-05-30T08:35:00.118Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T08:35:01.788Z\",\r\n \"Message\": \"Execute Query: Object reference not set to an instance of an object.\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, CommandType commandType, Dictionary`2 parameters, Int32 timeout)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T08:35:01.804Z\",\r\n \"Message\": \"Execute Query: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at DatabaseTransaction \\\"Start Transaction\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234\",\r\n \"StartTime\": \"2026-05-30T08:35:00.331Z\",\r\n \"EndTime\": \"2026-05-30T08:35:01.842Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"Execute Query: Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at DatabaseTransaction \\\"Start Transaction\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, CommandType commandType, Dictionary`2 parameters, Int32 timeout)\\r\\n at UiPath.Database.Activities.ExecuteQuery.ExecuteInternal(AsyncCodeActivityContext context)\\r\\n at UiPath.Database.Activities.ExecuteQuery.BeginExecute(AsyncCodeActivityContext context, AsyncCallback callback, Object state)\",\r\n \"CreationTime\": \"2026-05-30T08:35:00.118Z\",\r\n \"ReleaseName\": \"OrderSettlementTxn\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5042118,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T08:35:01.842Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234\",\r\n \"Id\": 671208842\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a6d9f7bc-8ca3-4fb0-91d2-6d7e8f901234", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-30T08:35:00.640Z\",\r\n \"Message\": \"OrderSettlementTxn execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-30T08:35:01.502Z\",\r\n \"Message\": \"Start Transaction: opened connection with ProviderName 'Microsoft.Data.SqlClient' and connection string 'Server=sqlsales02;Database=Orders;Integrated Security=true;Encrypt=true'; transaction begun, connection assigned to variable 'dbTxn'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-30T08:35:01.731Z\",\r\n \"Message\": \"Execute Query: executing with ExistingDbConnection = (null) [property is empty - not bound to 'dbTxn']; Sql = 'SELECT OrderId, Status FROM Orders WHERE Status = 'Pending''\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T08:35:01.788Z\",\r\n \"Message\": \"Execute Query: Object reference not set to an instance of an object.\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Database.DatabaseConnection.ExecuteQuery(String sql, CommandType commandType, Dictionary`2 parameters, Int32 timeout)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T08:35:01.804Z\",\r\n \"Message\": \"Execute Query: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ExecuteQuery \\\"Execute Query\\\"\\r\\n at DatabaseTransaction \\\"Start Transaction\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f5c8e6ab-7b92-4eaf-90c1-5c6d7e8f9012\",\r\n \"Name\": \"Sales Operations\",\r\n \"Path\": \"Sales Operations\",\r\n \"Description\": \"Folder hosting the order settlement and reconciliation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c4f1a983-6b2d-4e57-9a1c-3d8e0f5b7a26\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/task.yaml index 4c3c5075f9..448ee507f0 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-connection-not-propagated/task.yaml @@ -30,22 +30,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/README.md new file mode 100644 index 0000000000..7eaa672020 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/README.md @@ -0,0 +1,81 @@ +# Start Transaction Failure — Provider Type-Initializer Crash After Migration + +This scenario reproduces a runtime `Start Transaction` failure that +begins right after a project is migrated from **Windows - Legacy** +(.NET Framework) to **Windows** (.NET 6+). The `Start Transaction` +(`DatabaseTransaction`) activity still uses +`ProviderName="System.Data.SqlClient"`, so opening the transaction +connection crashes when the provider type initializes: +`System.TypeInitializationException: The type initializer for +'Microsoft.Data.SqlClient.SqlConnection' threw an exception`, which +Orchestrator surfaces as `Start Transaction: A database error occurred`. +The fault is at the Start Transaction connect step — before any child +activity in the scope runs — so it is unmistakably the Start Transaction +activity. + +## What this scenario uncovers + +**Root Cause:** The project's `targetFramework` is `Windows` (migrated), +but the `Start Transaction` activity was never updated off the legacy +provider. On .NET 6+ the `System.Data.SqlClient` provider's type +initializer fails, so the transaction connection never opens. The fix is +to **switch the Start Transaction provider to `Microsoft.Data.SqlClient`** +and confirm the driver is installed on the Robot host. + +This maps to: +`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` +(BRANCH 3 — post-migration provider / type breakage). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — `targetFramework: "Windows"`, a `Start Transaction` using `System.Data.SqlClient` wrapping a well-formed parameterized `Execute Non Query` that never runs | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Finance Operations` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `InvoicePostingTxn` job. +3. `uip or jobs get --output json` → `Info` carries `Start Transaction: A database error occurred` wrapping `System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception`, with the stack located at `DatabaseTransaction "Start Transaction"` (no child activity ran). +4. `uip or jobs logs --output json` → a Trace line showing the transaction connect attempt with `ProviderName 'System.Data.SqlClient'`, immediately followed by the type-initializer Error. +5. `process/project.json` → `targetFramework: "Windows"` (migrated); `process/Main.xaml` → `Start Transaction` with `ProviderName="System.Data.SqlClient"`. + +## How this differs from sibling branches + +| Dimension | branch 1 (silent rollback) | branch 2 (body-skip bug) | branch 3 (provider migration) (this) | branch 4 (connection not propagated) | +|---|---|---|---|---| +| Job state | **Success** (wrong) | **Success** (empty) | **Faulted** | Faulted (or non-transactional writes) | +| Where the fault is | swallowed child error / no rollback | package version `v1.5.0` | **Start Transaction provider/type after migration** | child `ExistingDbConnection` wiring | +| Error signature | none (green job, partial data) | none (green job, no child logs) | **`Microsoft.Data.SqlClient` type-initializer crash at connect** | `NullReferenceException` / instant `Timeout expired` | +| Fix | re-raise in catch so the scope rolls back | move off the defective package build | **switch Start Transaction to Microsoft.Data.SqlClient + driver on host** | thread the scope's `DatabaseConnection` into every child | + +The failing layer here is the **provider / type-initializer breakage on +the Start Transaction activity after migration**, raised at the connect +step before any child runs — not a swallowed rollback, not a package +body-skip, not the child connection wiring, not SQL, not a timeout, and +not the database. The connection string itself is well-formed; the fault +is the legacy PROVIDER failing to initialize on the modern runtime. The +agent must recommend switching to `Microsoft.Data.SqlClient` and +confirming the driver on the Robot host. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the start-transaction-failures playbook (branch 3) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) name the `Microsoft.Data.SqlClient` type-initializer crash, (b) attribute it to the Start Transaction activity still using `System.Data.SqlClient` after the project was migrated to Windows, and (c) recommend switching the provider to `Microsoft.Data.SqlClient` and confirming the driver is installed on the Robot host. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name start-transaction-provider-migration --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/08cd4b790c.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/08cd4b790c.json deleted file mode 100644 index f85f02f89d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/08cd4b790c.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90", - "StartTime": "2026-05-30T06:20:00.418Z", - "EndTime": "2026-05-30T06:20:01.503Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f2a3b4c5-6d7e-4f8a-9b0c-1d2e3f4a5b6c", - "Info": "Start Transaction: A database error occurred.\r\n\r\nin Main.xaml\r\n at DatabaseTransaction \"Start Transaction\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Database.Activities.DatabaseException: Start Transaction: A database error occurred. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnectionFactory' threw an exception. ---> System.PlatformNotSupportedException: System.Data.SqlClient is not supported on this platform.\r\n at Microsoft.Data.SqlClient.SqlConnectionFactory.cctor()\r\n --- End of inner exception stack trace ---\r\n at Microsoft.Data.SqlClient.SqlConnection.cctor()\r\n --- End of inner exception stack trace ---\r\n at Microsoft.Data.SqlClient.SqlConnection.ctor(String connectionString)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\r\n at UiPath.Database.DatabaseTransaction.BeginTransaction(String connectionString, String provider, Int32 timeout)\r\n --- End of inner exception stack trace ---", - "CreationTime": "2026-05-30T06:20:00.221Z", - "ReleaseName": "InvoicePostingTxn", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5061902, - "OrganizationUnitId": 7791624, - "FolderKey": "d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T06:20:01.503Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90", - "Id": 671311904 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/8bbecf54cd.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/8bbecf54cd.json deleted file mode 100644 index 3e373256a0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/8bbecf54cd.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f", - "Name": "Finance Operations", - "Path": "Finance Operations", - "Description": "Folder hosting the invoice posting and reconciliation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b1e2f3a4-5c6d-4e7f-8a9b-0c1d2e3f4a5b", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/935971866d.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/935971866d.json deleted file mode 100644 index adaf49a13a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/935971866d.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-30T06:20:00.730Z", - "Message": "InvoicePostingTxn execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-30T06:20:01.388Z", - "Message": "Start Transaction: opening transaction connection with ProviderName 'System.Data.SqlClient' and connection string 'Server=sqlfin01;Database=Invoices;Integrated Security=SSPI;Encrypt=true'" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T06:20:01.441Z", - "Message": "Start Transaction: A database error occurred.\r\nSystem.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\r\n at Microsoft.Data.SqlClient.SqlConnection.ctor(String connectionString)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\r\n at UiPath.Database.DatabaseTransaction.BeginTransaction(String connectionString, String provider, Int32 timeout)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T06:20:01.460Z", - "Message": "Start Transaction: A database error occurred.\r\nin Main.xaml\r\n at DatabaseTransaction \"Start Transaction\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/README.md deleted file mode 100644 index b54eed758e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/README.md +++ /dev/null @@ -1,80 +0,0 @@ -# Start Transaction Failure — Provider Type-Initializer Crash After Migration - -This scenario reproduces a runtime `Start Transaction` failure that -begins right after a project is migrated from **Windows - Legacy** -(.NET Framework) to **Windows** (.NET 6+). The `Start Transaction` -(`DatabaseTransaction`) activity still uses -`ProviderName="System.Data.SqlClient"`, so opening the transaction -connection crashes when the provider type initializes: -`System.TypeInitializationException: The type initializer for -'Microsoft.Data.SqlClient.SqlConnection' threw an exception`, which -Orchestrator surfaces as `Start Transaction: A database error occurred`. -The fault is at the Start Transaction connect step — before any child -activity in the scope runs — so it is unmistakably the Start Transaction -activity. - -## What this scenario uncovers - -**Root Cause:** The project's `targetFramework` is `Windows` (migrated), -but the `Start Transaction` activity was never updated off the legacy -provider. On .NET 6+ the `System.Data.SqlClient` provider's type -initializer fails, so the transaction connection never opens. The fix is -to **switch the Start Transaction provider to `Microsoft.Data.SqlClient`** -and confirm the driver is installed on the Robot host. - -This maps to: -`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` -(BRANCH 3 — post-migration provider / type breakage). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — `targetFramework: "Windows"`, a `Start Transaction` using `System.Data.SqlClient` wrapping a well-formed parameterized `Execute Non Query` that never runs | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-3 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Finance Operations` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → the single faulted `InvoicePostingTxn` job. -3. `uip or jobs get --output json` → `Info` carries `Start Transaction: A database error occurred` wrapping `System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception`, with the stack located at `DatabaseTransaction "Start Transaction"` (no child activity ran). -4. `uip or jobs logs --output json` → a Trace line showing the transaction connect attempt with `ProviderName 'System.Data.SqlClient'`, immediately followed by the type-initializer Error. -5. `process/project.json` → `targetFramework: "Windows"` (migrated); `process/Main.xaml` → `Start Transaction` with `ProviderName="System.Data.SqlClient"`. - -## How this differs from sibling branches - -| Dimension | branch 1 (silent rollback) | branch 2 (body-skip bug) | branch 3 (provider migration) (this) | branch 4 (connection not propagated) | -|---|---|---|---|---| -| Job state | **Success** (wrong) | **Success** (empty) | **Faulted** | Faulted (or non-transactional writes) | -| Where the fault is | swallowed child error / no rollback | package version `v1.5.0` | **Start Transaction provider/type after migration** | child `ExistingDbConnection` wiring | -| Error signature | none (green job, partial data) | none (green job, no child logs) | **`Microsoft.Data.SqlClient` type-initializer crash at connect** | `NullReferenceException` / instant `Timeout expired` | -| Fix | re-raise in catch so the scope rolls back | move off the defective package build | **switch Start Transaction to Microsoft.Data.SqlClient + driver on host** | thread the scope's `DatabaseConnection` into every child | - -The failing layer here is the **provider / type-initializer breakage on -the Start Transaction activity after migration**, raised at the connect -step before any child runs — not a swallowed rollback, not a package -body-skip, not the child connection wiring, not SQL, not a timeout, and -not the database. The connection string itself is well-formed; the fault -is the legacy PROVIDER failing to initialize on the modern runtime. The -agent must recommend switching to `Microsoft.Data.SqlClient` and -confirming the driver on the Robot host. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the start-transaction-failures playbook (branch 3) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) name the `Microsoft.Data.SqlClient` type-initializer crash, (b) attribute it to the Start Transaction activity still using `System.Data.SqlClient` after the project was migrated to Windows, and (c) recommend switching the provider to `Microsoft.Data.SqlClient` and confirming the driver is installed on the Robot host. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name start-transaction-provider-migration --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/cd27f132ef.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/cd27f132ef.json deleted file mode 100644 index df70537bd6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/cd27f132ef.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-30T06:20:01.441Z", - "Message": "Start Transaction: A database error occurred.\r\nSystem.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\r\n at Microsoft.Data.SqlClient.SqlConnection.ctor(String connectionString)\r\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\r\n at UiPath.Database.DatabaseTransaction.BeginTransaction(String connectionString, String provider, Int32 timeout)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T06:20:01.460Z", - "Message": "Start Transaction: A database error occurred.\r\nin Main.xaml\r\n at DatabaseTransaction \"Start Transaction\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/eccb1e2c4b.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/eccb1e2c4b.json deleted file mode 100644 index 67c3d2f934..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/eccb1e2c4b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90", - "State": "Faulted", - "ProcessName": "InvoicePostingTxn", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-30T06:20:00.418Z", - "EndTime": "2026-05-30T06:20:01.503Z", - "CreationTime": "2026-05-30T06:20:00.221Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/manifest.json deleted file mode 100644 index 411dd0e11d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Start Transaction scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "8bbecf54cd.json" - }, - { - "match": "or jobs list --folder-key \"d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f\" --state Faulted --output json", - "file": "eccb1e2c4b.json" - }, - { - "match": "or jobs list --folder-key d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f --state Faulted --output json", - "file": "eccb1e2c4b.json" - }, - { - "match": "or jobs get \"e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90\" --output json", - "file": "08cd4b790c.json" - }, - { - "match": "or jobs get e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90 --output json", - "file": "08cd4b790c.json" - }, - { - "match": "or jobs logs \"e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90\" --level Error --output json", - "file": "cd27f132ef.json" - }, - { - "match": "or jobs logs e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90 --level Error --output json", - "file": "cd27f132ef.json" - }, - { - "match": "or jobs logs \"e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90\" --output json", - "file": "935971866d.json" - }, - { - "match": "or jobs logs e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90 --output json", - "file": "935971866d.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/uip-fixture.json new file mode 100644 index 0000000000..365358626d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoicePostingTxn\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-30T06:20:00.418Z\",\r\n \"EndTime\": \"2026-05-30T06:20:01.503Z\",\r\n \"CreationTime\": \"2026-05-30T06:20:00.221Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T06:20:01.441Z\",\r\n \"Message\": \"Start Transaction: A database error occurred.\\r\\nSystem.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\\r\\n at Microsoft.Data.SqlClient.SqlConnection.ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\\r\\n at UiPath.Database.DatabaseTransaction.BeginTransaction(String connectionString, String provider, Int32 timeout)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T06:20:01.460Z\",\r\n \"Message\": \"Start Transaction: A database error occurred.\\r\\nin Main.xaml\\r\\n at DatabaseTransaction \\\"Start Transaction\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90\",\r\n \"StartTime\": \"2026-05-30T06:20:00.418Z\",\r\n \"EndTime\": \"2026-05-30T06:20:01.503Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f2a3b4c5-6d7e-4f8a-9b0c-1d2e3f4a5b6c\",\r\n \"Info\": \"Start Transaction: A database error occurred.\\r\\n\\r\\nin Main.xaml\\r\\n at DatabaseTransaction \\\"Start Transaction\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Database.Activities.DatabaseException: Start Transaction: A database error occurred. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception. ---> System.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnectionFactory' threw an exception. ---> System.PlatformNotSupportedException: System.Data.SqlClient is not supported on this platform.\\r\\n at Microsoft.Data.SqlClient.SqlConnectionFactory.cctor()\\r\\n --- End of inner exception stack trace ---\\r\\n at Microsoft.Data.SqlClient.SqlConnection.cctor()\\r\\n --- End of inner exception stack trace ---\\r\\n at Microsoft.Data.SqlClient.SqlConnection.ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\\r\\n at UiPath.Database.DatabaseTransaction.BeginTransaction(String connectionString, String provider, Int32 timeout)\\r\\n --- End of inner exception stack trace ---\",\r\n \"CreationTime\": \"2026-05-30T06:20:00.221Z\",\r\n \"ReleaseName\": \"InvoicePostingTxn\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5061902,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T06:20:01.503Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90\",\r\n \"Id\": 671311904\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e4b7d5fa-6a81-4d9e-8fb0-4b5c6d7e8f90", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-30T06:20:00.730Z\",\r\n \"Message\": \"InvoicePostingTxn execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-30T06:20:01.388Z\",\r\n \"Message\": \"Start Transaction: opening transaction connection with ProviderName 'System.Data.SqlClient' and connection string 'Server=sqlfin01;Database=Invoices;Integrated Security=SSPI;Encrypt=true'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T06:20:01.441Z\",\r\n \"Message\": \"Start Transaction: A database error occurred.\\r\\nSystem.TypeInitializationException: The type initializer for 'Microsoft.Data.SqlClient.SqlConnection' threw an exception.\\r\\n at Microsoft.Data.SqlClient.SqlConnection.ctor(String connectionString)\\r\\n at UiPath.Database.DatabaseConnection.Connect(String connectionString, String provider, Int32 timeout)\\r\\n at UiPath.Database.DatabaseTransaction.BeginTransaction(String connectionString, String provider, Int32 timeout)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T06:20:01.460Z\",\r\n \"Message\": \"Start Transaction: A database error occurred.\\r\\nin Main.xaml\\r\\n at DatabaseTransaction \\\"Start Transaction\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d3a6c4e9-5f70-4c8d-9eaf-3a4b5c6d7e8f\",\r\n \"Name\": \"Finance Operations\",\r\n \"Path\": \"Finance Operations\",\r\n \"Description\": \"Folder hosting the invoice posting and reconciliation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b1e2f3a4-5c6d-4e7f-8a9b-0c1d2e3f4a5b\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/task.yaml index 0c1f068116..afd76d931f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-provider-migration/task.yaml @@ -34,22 +34,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/README.md new file mode 100644 index 0000000000..16a57c4ea7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/README.md @@ -0,0 +1,86 @@ +# Start Transaction Failure — Silent Success / No Rollback + +This scenario reproduces a `Start Transaction` failure where the **job +reports Success** even though the work it should have committed is missing +or incomplete. A child `Execute Non Query` inside the `Start Transaction` +scope faults on a FOREIGN KEY violation, but an in-scope `Try Catch` +catches `System.Exception` and **only logs** a Warn message (no `Throw`). +The fault never propagates out of the scope, so the transaction does not +roll back: the scope exits normally, **commits** the partial work (the +header insert), and the job ends **Successful** on incomplete data. + +## What this scenario uncovers + +**Root Cause:** The in-scope catch swallows the child fault, so the +all-or-nothing guarantee of the transaction is lost. The header row +commits, the detail rows are never written, and Orchestrator records a +clean Success — the bug is exactly that it looks green. The fix is to +**re-raise (`Throw`)** in the catch so the fault leaves the `Start +Transaction` scope and the transaction **rolls back** (and the job faults +visibly) — not log-and-continue. + +This maps to: +`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` +(BRANCH 1 — silent success / no rollback). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — a `Start Transaction` (`ui:DatabaseTransaction`, `Microsoft.Data.SqlClient`) scope wrapping a `Try Catch` over two `Execute Non Query` inserts (both bound to the scope's `[dbTxn]`); the `Catch` for `System.Exception` holds only a `Log Message` (Warn) with no `Throw` | + +The decisive evidence chain: + +1. `uip or folders list --output json` → the `Finance Operations` folder. +2. `uip or jobs list --folder-key --state Faulted --output json` → **empty** list (no faulted job; the failure hides behind a green Success). +3. `uip or jobs list --folder-key ` (no state) / `--state Successful` → the single **Successful** `BatchLedgerPost` job. +4. `uip or jobs get --output json` → `State: "Successful"`, benign `Info: "Execution ended."`. +5. `uip or jobs logs --output json` → the smoking gun: the header insert succeeds, the detail insert logs an **Error** (FK violation), an **Info** "caught and continuing" line immediately follows, then the scope **commits** and the job ends Successful — the child error was swallowed and the transaction still committed. +6. `process/Main.xaml` → the in-scope `Try Catch` whose `Catch` contains only a `Log Message` (Warn) with no `Throw`. + +## How this differs from sibling branches + +| Dimension | branch 1 (silent rollback) (this) | branch 2 (package body-skip) | branch 3 (provider mismatch) | branch 4 (connection not propagated) | +|---|---|---|---|---| +| Job state | **Successful (false green)** | Successful | Faulted | Faulted (or non-transactional writes) | +| Child activities run? | **yes — and one errored** | no (body skipped) | n/a (fails at connect) | yes | +| Signature | **child Error + "caught, continuing" Info, then commit** | empty scope, no logs, on v1.5.0 | `Keyword not supported` / type-initializer crash | `Object reference not set` / instant `Timeout expired`, or auto-committed writes | +| Where the fault is | **swallowed in-scope catch (no Throw)** | package version | provider/driver after migration | child `ExistingDbConnection` wiring | +| Fix | **re-raise (Throw) so the transaction rolls back** | change package version | switch to `Microsoft.Data.SqlClient` | thread the scope's `DatabaseConnection` into every child | + +The failing layer here is the **exception flow inside the transaction +scope** — a child error swallowed by a no-`Throw` catch — not a data +problem, a timeout, the connection wiring, the provider, the package +version, or the database. The agent must recommend **re-raising (`Throw`)** +in the catch so the transaction rolls back — not blame the data, a timeout, +a null connection, a provider mismatch, a package bug, or a DB outage. + +## Synthetic fixtures + +documented branch-1 signature, not captured from a real +`.local/investigations/` run. They are internally consistent (one +Successful job, an empty Faulted list, and job logs carrying the +discovery chain. Replace them with verbatim captures from a real +investigation before treating this scenario as a hard regression signal. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the start-transaction-failures playbook (branch 1) AND reached the same root cause as `RESOLUTION.md`. +- Conclusion must (a) identify that a child activity inside the `Start Transaction` scope failed and the error was swallowed by an in-scope catch with no `Throw`, (b) explain that the transaction committed / did not roll back so the job reports Success on incomplete data, and (c) recommend re-raising (`Throw`) in the catch so the transaction rolls back. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name start-transaction-silent-rollback --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/20f04b8eac.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/20f04b8eac.json deleted file mode 100644 index af417b1183..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/20f04b8eac.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d", - "Name": "Finance Operations", - "Path": "Finance Operations", - "Description": "Folder hosting the ledger posting and reconciliation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a7c9e1b3-5d7f-4a2c-8b6d-9e0f1a2b3c4d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/34b6d8b8eb.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/34b6d8b8eb.json deleted file mode 100644 index 3bc37ac9fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/34b6d8b8eb.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e", - "StartTime": "2026-05-30T02:10:01.118Z", - "EndTime": "2026-05-30T02:10:06.402Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d3f6c4e9-5f7a-4c8d-9e0f-3a4b5c6d7e8f", - "Info": "Execution ended.", - "CreationTime": "2026-05-30T02:10:00.904Z", - "ReleaseName": "BatchLedgerPost", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5048217, - "OrganizationUnitId": 7791624, - "FolderKey": "b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T02:10:06.402Z", - "JobError": {}, - "ErrorCode": "", - "TraceId": "c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e", - "Id": 671320514 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/5f66406f9d.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/5f66406f9d.json deleted file mode 100644 index fde5d56f5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/5f66406f9d.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-30T02:10:01.402Z", - "Message": "BatchLedgerPost execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-30T02:10:02.118Z", - "Message": "Start Transaction: opening connection with ProviderName 'Microsoft.Data.SqlClient' and beginning a database transaction on 'Server=sqlfin01;Database=Ledger;Integrated Security=true;Encrypt=true'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-30T02:10:02.704Z", - "Message": "Execute Non Query: inserted header row for batch 'B-20260530-0210' into LedgerBatchHeader (1 row affected)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-30T02:10:03.221Z", - "Message": "Execute Non Query: A database error occurred.\r\nMicrosoft.Data.SqlClient.SqlException (0x80131904): The INSERT statement conflicted with the FOREIGN KEY constraint \"FK_LedgerBatchDetail_Account\". The conflict occurred in database \"Ledger\", table \"dbo.Account\", column 'AccountId'.\r\n at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)\r\n at UiPath.Database.DatabaseConnection.ExecuteNonQuery(String sql, Dictionary`2 parameters, Int32 timeout)" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-30T02:10:03.405Z", - "Message": "Detail insert failed, caught and continuing: The INSERT statement conflicted with the FOREIGN KEY constraint \"FK_LedgerBatchDetail_Account\"." - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-30T02:10:05.880Z", - "Message": "Start Transaction: scope exited normally; committing database transaction" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-30T02:10:06.291Z", - "Message": "BatchLedgerPost execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/README.md deleted file mode 100644 index 814b80219f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/README.md +++ /dev/null @@ -1,87 +0,0 @@ -# Start Transaction Failure — Silent Success / No Rollback - -This scenario reproduces a `Start Transaction` failure where the **job -reports Success** even though the work it should have committed is missing -or incomplete. A child `Execute Non Query` inside the `Start Transaction` -scope faults on a FOREIGN KEY violation, but an in-scope `Try Catch` -catches `System.Exception` and **only logs** a Warn message (no `Throw`). -The fault never propagates out of the scope, so the transaction does not -roll back: the scope exits normally, **commits** the partial work (the -header insert), and the job ends **Successful** on incomplete data. - -## What this scenario uncovers - -**Root Cause:** The in-scope catch swallows the child fault, so the -all-or-nothing guarantee of the transaction is lost. The header row -commits, the detail rows are never written, and Orchestrator records a -clean Success — the bug is exactly that it looks green. The fix is to -**re-raise (`Throw`)** in the catch so the fault leaves the `Start -Transaction` scope and the transaction **rolls back** (and the job faults -visibly) — not log-and-continue. - -This maps to: -`references/activity-packages/database-activities/playbooks/start-transaction-failures.md` -(BRANCH 1 — silent success / no rollback). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — a `Start Transaction` (`ui:DatabaseTransaction`, `Microsoft.Data.SqlClient`) scope wrapping a `Try Catch` over two `Execute Non Query` inserts (both bound to the scope's `[dbTxn]`); the `Catch` for `System.Exception` holds only a `Log Message` (Warn) with no `Throw` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented branch-1 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence chain: - -1. `uip or folders list --output json` → the `Finance Operations` folder. -2. `uip or jobs list --folder-key --state Faulted --output json` → **empty** list (no faulted job; the failure hides behind a green Success). -3. `uip or jobs list --folder-key ` (no state) / `--state Successful` → the single **Successful** `BatchLedgerPost` job. -4. `uip or jobs get --output json` → `State: "Successful"`, benign `Info: "Execution ended."`. -5. `uip or jobs logs --output json` → the smoking gun: the header insert succeeds, the detail insert logs an **Error** (FK violation), an **Info** "caught and continuing" line immediately follows, then the scope **commits** and the job ends Successful — the child error was swallowed and the transaction still committed. -6. `process/Main.xaml` → the in-scope `Try Catch` whose `Catch` contains only a `Log Message` (Warn) with no `Throw`. - -## How this differs from sibling branches - -| Dimension | branch 1 (silent rollback) (this) | branch 2 (package body-skip) | branch 3 (provider mismatch) | branch 4 (connection not propagated) | -|---|---|---|---|---| -| Job state | **Successful (false green)** | Successful | Faulted | Faulted (or non-transactional writes) | -| Child activities run? | **yes — and one errored** | no (body skipped) | n/a (fails at connect) | yes | -| Signature | **child Error + "caught, continuing" Info, then commit** | empty scope, no logs, on v1.5.0 | `Keyword not supported` / type-initializer crash | `Object reference not set` / instant `Timeout expired`, or auto-committed writes | -| Where the fault is | **swallowed in-scope catch (no Throw)** | package version | provider/driver after migration | child `ExistingDbConnection` wiring | -| Fix | **re-raise (Throw) so the transaction rolls back** | change package version | switch to `Microsoft.Data.SqlClient` | thread the scope's `DatabaseConnection` into every child | - -The failing layer here is the **exception flow inside the transaction -scope** — a child error swallowed by a no-`Throw` catch — not a data -problem, a timeout, the connection wiring, the provider, the package -version, or the database. The agent must recommend **re-raising (`Throw`)** -in the catch so the transaction rolls back — not blame the data, a timeout, -a null connection, a provider mismatch, a package bug, or a DB outage. - -## Synthetic fixtures - -The `fixtures/mocks/responses/*.json` files are **hand-authored** from the -documented branch-1 signature, not captured from a real -`.local/investigations/` run. They are internally consistent (one -Successful job, an empty Faulted list, and job logs carrying the -swallowed-error smoking gun) so the mock dispatcher resolves the full -discovery chain. Replace them with verbatim captures from a real -investigation before treating this scenario as a hard regression signal. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the start-transaction-failures playbook (branch 1) AND reached the same root cause as `RESOLUTION.md`. -- Conclusion must (a) identify that a child activity inside the `Start Transaction` scope failed and the error was swallowed by an in-scope catch with no `Throw`, (b) explain that the transaction committed / did not roll back so the job reports Success on incomplete data, and (c) recommend re-raising (`Throw`) in the catch so the transaction rolls back. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name start-transaction-silent-rollback --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/c8a70192d7.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/c8a70192d7.json deleted file mode 100644 index 02fd432175..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/c8a70192d7.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e", - "State": "Successful", - "ProcessName": "BatchLedgerPost", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-30T02:10:01.118Z", - "EndTime": "2026-05-30T02:10:06.402Z", - "CreationTime": "2026-05-30T02:10:00.904Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/manifest.json deleted file mode 100644 index 2b3c928fd8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Start Transaction scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "20f04b8eac.json" - }, - { - "match": "or jobs list --folder-key \"b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d\" --state Successful --output json", - "file": "c8a70192d7.json" - }, - { - "match": "or jobs list --folder-key b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d --state Successful --output json", - "file": "c8a70192d7.json" - }, - { - "match": "or jobs list --folder-key \"b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d\" --output json", - "file": "c8a70192d7.json" - }, - { - "match": "or jobs list --folder-key b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d --output json", - "file": "c8a70192d7.json" - }, - { - "match": "or jobs get \"c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e\" --output json", - "file": "34b6d8b8eb.json" - }, - { - "match": "or jobs get c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e --output json", - "file": "34b6d8b8eb.json" - }, - { - "match": "or jobs logs \"c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e\" --output json", - "file": "5f66406f9d.json" - }, - { - "match": "or jobs logs c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e --output json", - "file": "5f66406f9d.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/uip-fixture.json new file mode 100644 index 0000000000..699e121204 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/data/uip-fixture.json @@ -0,0 +1,93 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BatchLedgerPost\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-30T02:10:01.118Z\",\r\n \"EndTime\": \"2026-05-30T02:10:06.402Z\",\r\n \"CreationTime\": \"2026-05-30T02:10:00.904Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BatchLedgerPost\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-30T02:10:01.118Z\",\r\n \"EndTime\": \"2026-05-30T02:10:06.402Z\",\r\n \"CreationTime\": \"2026-05-30T02:10:00.904Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e\",\r\n \"StartTime\": \"2026-05-30T02:10:01.118Z\",\r\n \"EndTime\": \"2026-05-30T02:10:06.402Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d3f6c4e9-5f7a-4c8d-9e0f-3a4b5c6d7e8f\",\r\n \"Info\": \"Execution ended.\",\r\n \"CreationTime\": \"2026-05-30T02:10:00.904Z\",\r\n \"ReleaseName\": \"BatchLedgerPost\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5048217,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T02:10:06.402Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e\",\r\n \"Id\": 671320514\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c2e5b3d8-4e6f-4b7c-8d9e-2f3a4b5c6d7e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-30T02:10:01.402Z\",\r\n \"Message\": \"BatchLedgerPost execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-30T02:10:02.118Z\",\r\n \"Message\": \"Start Transaction: opening connection with ProviderName 'Microsoft.Data.SqlClient' and beginning a database transaction on 'Server=sqlfin01;Database=Ledger;Integrated Security=true;Encrypt=true'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-30T02:10:02.704Z\",\r\n \"Message\": \"Execute Non Query: inserted header row for batch 'B-20260530-0210' into LedgerBatchHeader (1 row affected)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-30T02:10:03.221Z\",\r\n \"Message\": \"Execute Non Query: A database error occurred.\\r\\nMicrosoft.Data.SqlClient.SqlException (0x80131904): The INSERT statement conflicted with the FOREIGN KEY constraint \\\"FK_LedgerBatchDetail_Account\\\". The conflict occurred in database \\\"Ledger\\\", table \\\"dbo.Account\\\", column 'AccountId'.\\r\\n at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)\\r\\n at UiPath.Database.DatabaseConnection.ExecuteNonQuery(String sql, Dictionary`2 parameters, Int32 timeout)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-30T02:10:03.405Z\",\r\n \"Message\": \"Detail insert failed, caught and continuing: The INSERT statement conflicted with the FOREIGN KEY constraint \\\"FK_LedgerBatchDetail_Account\\\".\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-30T02:10:05.880Z\",\r\n \"Message\": \"Start Transaction: scope exited normally; committing database transaction\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-30T02:10:06.291Z\",\r\n \"Message\": \"BatchLedgerPost execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b1f4a2c7-3d5e-4a6b-9c8d-1e2f3a4b5c6d\",\r\n \"Name\": \"Finance Operations\",\r\n \"Path\": \"Finance Operations\",\r\n \"Description\": \"Folder hosting the ledger posting and reconciliation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a7c9e1b3-5d7f-4a2c-8b6d-9e0f1a2b3c4d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/task.yaml index d4cde02d98..610a99f33e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/db-start-transaction-silent-rollback/task.yaml @@ -29,22 +29,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/README.md new file mode 100644 index 0000000000..d305e982c8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/README.md @@ -0,0 +1,40 @@ +# Download File from URL Failure - HTTP 403 (Forbidden) + +This scenario reproduces a `Download File from URL` failure where the server +returns `403 (Forbidden)`. The activity issues a native HTTP request that carries +no authentication / cookies / browser session, so a portal-gated URL rejects it — +even though the same link downloads in a logged-in browser. + +## What this scenario uncovers + +**Root Cause:** The native download has no auth/session; the portal-gated URL +returns 403. + +This maps to: +`references/activity-packages/file-operations/playbooks/download-file-403-401-auth.md` + +"Works in my logged-in browser, 403 for the robot" is the tell. The fix is to +authenticate / set `UserAgentHeader`, or download via an authenticated UI browser +session (Click + Wait for Download) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a single `Download File from URL` to a portal URL (no `UserAgentHeader`/auth) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies that the native download + carries no authentication for the portal-gated URL (403) and recommends + authenticating / setting `UserAgentHeader`, or downloading via an authenticated + UI browser session (Click + Wait for Download), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/README.md deleted file mode 100644 index 02ebe2f96e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Download File from URL Failure - HTTP 403 (Forbidden) - -This scenario reproduces a `Download File from URL` failure where the server -returns `403 (Forbidden)`. The activity issues a native HTTP request that carries -no authentication / cookies / browser session, so a portal-gated URL rejects it — -even though the same link downloads in a logged-in browser. - -## What this scenario uncovers - -**Root Cause:** The native download has no auth/session; the portal-gated URL -returns 403. - -This maps to: -`references/activity-packages/file-operations/playbooks/download-file-403-401-auth.md` - -"Works in my logged-in browser, 403 for the robot" is the tell. The fix is to -authenticate / set `UserAgentHeader`, or download via an authenticated UI browser -session (Click + Wait for Download) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a single `Download File from URL` to a portal URL (no `UserAgentHeader`/auth) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies that the native download - carries no authentication for the portal-gated URL (403) and recommends - authenticating / setting `UserAgentHeader`, or downloading via an authenticated - UI browser session (Click + Wait for Download), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/manifest.json deleted file mode 100644 index 23dd450807..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"fa020002-d4e5-4f60-8a02-000000000002\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa020002-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key fa020002-d4e5-4f60-8a02-000000000002 --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa020002-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"fb020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-get-fb020002-output-json.json" - }, - { - "match": "or jobs get fb020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-get-fb020002-output-json.json" - }, - { - "match": "or jobs logs \"fb020002-e5f6-4071-9b02-000000000002\" --level Error --output json", - "file": "or-jobs-logs-fb020002-level-error-output-json.json" - }, - { - "match": "or jobs logs fb020002-e5f6-4071-9b02-000000000002 --level Error --output json", - "file": "or-jobs-logs-fb020002-level-error-output-json.json" - }, - { - "match": "or jobs logs \"fb020002-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-logs-fb020002-output-json.json" - }, - { - "match": "or jobs logs fb020002-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-logs-fb020002-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 9bb95fa52c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "fa020002-d4e5-4f60-8a02-000000000002", - "Name": "Portal Sync", - "Path": "Portal Sync", - "Description": "Production folder for portal file-sync processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-get-fb020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-get-fb020002-output-json.json deleted file mode 100644 index fe2d35867f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-get-fb020002-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "fb020002-e5f6-4071-9b02-000000000002", - "StartTime": "2026-06-16T10:02:19.140Z", - "EndTime": "2026-06-16T10:02:21.480Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "fc020002-a7b8-4192-9c02-000000000002", - "Info": "Download File from URL: The remote server returned an error: (403) Forbidden.\r\n\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Http.HttpRequestException: The remote server returned an error: (403) Forbidden. (https://portal.example.com/reports/q3-summary.xlsx)", - "CreationTime": "2026-06-16T10:02:18.700Z", - "ReleaseName": "PortalFetch", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7016002, - "OrganizationUnitId": 7791618, - "FolderKey": "fa020002-d4e5-4f60-8a02-000000000002", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T10:02:21.480Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "fb020002-e5f6-4071-9b02-000000000002", - "Id": 716010202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-list-folder-key-fa020002-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-list-folder-key-fa020002-state-faulted-output-json.json deleted file mode 100644 index 6f2b58a33c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-list-folder-key-fa020002-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "fb020002-e5f6-4071-9b02-000000000002", - "State": "Faulted", - "ProcessName": "PortalFetch", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T10:02:19.140Z", - "EndTime": "2026-06-16T10:02:21.480Z", - "CreationTime": "2026-06-16T10:02:18.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-logs-fb020002-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-logs-fb020002-level-error-output-json.json deleted file mode 100644 index 455c72b343..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-logs-fb020002-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T10:02:21.330Z", - "Message": "[Download File from URL] The remote server returned an error: (403) Forbidden. (https://portal.example.com/reports/q3-summary.xlsx)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T10:02:21.400Z", - "Message": "Download File from URL: The remote server returned an error: (403) Forbidden.\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-logs-fb020002-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-logs-fb020002-output-json.json deleted file mode 100644 index fa2ca13b4b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/m/r/or-jobs-logs-fb020002-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T10:02:19.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T10:02:20.260Z", - "Message": "[Download File from URL] Requesting https://portal.example.com/reports/q3-summary.xlsx" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T10:02:21.330Z", - "Message": "[Download File from URL] The remote server returned an error: (403) Forbidden. (https://portal.example.com/reports/q3-summary.xlsx)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T10:02:21.400Z", - "Message": "Download File from URL: The remote server returned an error: (403) Forbidden.\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T10:02:21.450Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/uip-fixture.json new file mode 100644 index 0000000000..4351fe1e83 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "fa020002-d4e5-4f60-8a02-000000000002", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fb020002-e5f6-4071-9b02-000000000002\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PortalFetch\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T10:02:19.140Z\",\r\n \"EndTime\": \"2026-06-16T10:02:21.480Z\",\r\n \"CreationTime\": \"2026-06-16T10:02:18.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb020002-e5f6-4071-9b02-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T10:02:21.330Z\",\r\n \"Message\": \"[Download File from URL] The remote server returned an error: (403) Forbidden. (https://portal.example.com/reports/q3-summary.xlsx)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T10:02:21.400Z\",\r\n \"Message\": \"Download File from URL: The remote server returned an error: (403) Forbidden.\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "fb020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"fb020002-e5f6-4071-9b02-000000000002\",\r\n \"StartTime\": \"2026-06-16T10:02:19.140Z\",\r\n \"EndTime\": \"2026-06-16T10:02:21.480Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"fc020002-a7b8-4192-9c02-000000000002\",\r\n \"Info\": \"Download File from URL: The remote server returned an error: (403) Forbidden.\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Http.HttpRequestException: The remote server returned an error: (403) Forbidden. (https://portal.example.com/reports/q3-summary.xlsx)\",\r\n \"CreationTime\": \"2026-06-16T10:02:18.700Z\",\r\n \"ReleaseName\": \"PortalFetch\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7016002,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"fa020002-d4e5-4f60-8a02-000000000002\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T10:02:21.480Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"fb020002-e5f6-4071-9b02-000000000002\",\r\n \"Id\": 716010202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb020002-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T10:02:19.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T10:02:20.260Z\",\r\n \"Message\": \"[Download File from URL] Requesting https://portal.example.com/reports/q3-summary.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T10:02:21.330Z\",\r\n \"Message\": \"[Download File from URL] The remote server returned an error: (403) Forbidden. (https://portal.example.com/reports/q3-summary.xlsx)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T10:02:21.400Z\",\r\n \"Message\": \"Download File from URL: The remote server returned an error: (403) Forbidden.\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T10:02:21.450Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fa020002-d4e5-4f60-8a02-000000000002\",\r\n \"Name\": \"Portal Sync\",\r\n \"Path\": \"Portal Sync\",\r\n \"Description\": \"Production folder for portal file-sync processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/task.yaml index d3d44f6746..aa792021c8 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-403-forbidden/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/README.md new file mode 100644 index 0000000000..e17cca3f90 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/README.md @@ -0,0 +1,43 @@ +# Download File from URL Failure - "Don't know about such a host" (DNS / Firewall) + +This scenario reproduces a `Download File from URL` name-resolution failure. The +activity faults with `Don't know about such a host` because DNS cannot resolve the +URL's host from the robot, or an enterprise firewall / SSL-inspection proxy blocks +the automated outbound connection — even though the URL opens from a user's own +laptop browser. + +## What this scenario uncovers + +**Root Cause:** The host doesn't resolve / isn't reachable from the **robot +machine** (DNS or firewall/proxy), not a workflow defect. + +This maps to: +`references/activity-packages/file-operations/playbooks/download-file-host-not-found.md` + +The discriminator vs a 401/403: this is a **host-resolution** failure (the request +never reaches the server), not an auth rejection. The fix is an environment/network +change (whitelist / verify reachability from the robot) — not a workflow or host +command run by the agent. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a single `Download File from URL` to an external partner host | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies a DNS / firewall / + SSL-inspection host-resolution failure from the robot (not an auth/401-403 + issue) and recommends the network fix — whitelist the endpoint for the robot and + verify the host resolves/reaches from the robot machine — without fabricating + host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/README.md deleted file mode 100644 index bc4d556bd3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Download File from URL Failure - "Don't know about such a host" (DNS / Firewall) - -This scenario reproduces a `Download File from URL` name-resolution failure. The -activity faults with `Don't know about such a host` because DNS cannot resolve the -URL's host from the robot, or an enterprise firewall / SSL-inspection proxy blocks -the automated outbound connection — even though the URL opens from a user's own -laptop browser. - -## What this scenario uncovers - -**Root Cause:** The host doesn't resolve / isn't reachable from the **robot -machine** (DNS or firewall/proxy), not a workflow defect. - -This maps to: -`references/activity-packages/file-operations/playbooks/download-file-host-not-found.md` - -The discriminator vs a 401/403: this is a **host-resolution** failure (the request -never reaches the server), not an auth rejection. The fix is an environment/network -change (whitelist / verify reachability from the robot) — not a workflow or host -command run by the agent. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a single `Download File from URL` to an external partner host | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies a DNS / firewall / - SSL-inspection host-resolution failure from the robot (not an auth/401-403 - issue) and recommends the network fix — whitelist the endpoint for the robot and - verify the host resolves/reaches from the robot machine — without fabricating - host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/manifest.json deleted file mode 100644 index 9c15ddc9c4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"fa040004-d4e5-4f60-8a04-000000000004\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa040004-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key fa040004-d4e5-4f60-8a04-000000000004 --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa040004-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"fb040004-e5f6-4071-9b04-000000000004\" --output json", - "file": "or-jobs-get-fb040004-output-json.json" - }, - { - "match": "or jobs get fb040004-e5f6-4071-9b04-000000000004 --output json", - "file": "or-jobs-get-fb040004-output-json.json" - }, - { - "match": "or jobs logs \"fb040004-e5f6-4071-9b04-000000000004\" --level Error --output json", - "file": "or-jobs-logs-fb040004-level-error-output-json.json" - }, - { - "match": "or jobs logs fb040004-e5f6-4071-9b04-000000000004 --level Error --output json", - "file": "or-jobs-logs-fb040004-level-error-output-json.json" - }, - { - "match": "or jobs logs \"fb040004-e5f6-4071-9b04-000000000004\" --output json", - "file": "or-jobs-logs-fb040004-output-json.json" - }, - { - "match": "or jobs logs fb040004-e5f6-4071-9b04-000000000004 --output json", - "file": "or-jobs-logs-fb040004-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 6fd062164f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "fa040004-d4e5-4f60-8a04-000000000004", - "Name": "External Feeds", - "Path": "External Feeds", - "Description": "Production folder for external feed-download processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-get-fb040004-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-get-fb040004-output-json.json deleted file mode 100644 index eb4e5c41ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-get-fb040004-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "fb040004-e5f6-4071-9b04-000000000004", - "StartTime": "2026-06-16T14:26:51.140Z", - "EndTime": "2026-06-16T14:26:53.300Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "fc040004-a7b8-4192-9c04-000000000004", - "Info": "Download File from URL: Don't know about such a host. (feeds.partner-portal.example:443)\r\n\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Http.HttpRequestException: Don't know about such a host. (feeds.partner-portal.example:443) ---> System.Net.Sockets.SocketException: Don't know about such a host.", - "CreationTime": "2026-06-16T14:26:50.700Z", - "ReleaseName": "FeedDownloader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7016004, - "OrganizationUnitId": 7791618, - "FolderKey": "fa040004-d4e5-4f60-8a04-000000000004", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T14:26:53.300Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "fb040004-e5f6-4071-9b04-000000000004", - "Id": 716010404 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-list-folder-key-fa040004-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-list-folder-key-fa040004-state-faulted-output-json.json deleted file mode 100644 index 75a74edbaa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-list-folder-key-fa040004-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "fb040004-e5f6-4071-9b04-000000000004", - "State": "Faulted", - "ProcessName": "FeedDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T14:26:51.140Z", - "EndTime": "2026-06-16T14:26:53.300Z", - "CreationTime": "2026-06-16T14:26:50.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-logs-fb040004-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-logs-fb040004-level-error-output-json.json deleted file mode 100644 index 6430a2816d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-logs-fb040004-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T14:26:53.150Z", - "Message": "[Download File from URL] Don't know about such a host. (feeds.partner-portal.example:443)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T14:26:53.220Z", - "Message": "Download File from URL: Don't know about such a host. (feeds.partner-portal.example:443)\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-logs-fb040004-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-logs-fb040004-output-json.json deleted file mode 100644 index 992a6fbcc3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/m/r/or-jobs-logs-fb040004-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T14:26:51.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T14:26:52.180Z", - "Message": "[Download File from URL] Resolving https://feeds.partner-portal.example/data/export.csv" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T14:26:53.150Z", - "Message": "[Download File from URL] Don't know about such a host. (feeds.partner-portal.example:443)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T14:26:53.220Z", - "Message": "Download File from URL: Don't know about such a host. (feeds.partner-portal.example:443)\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T14:26:53.270Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..1dec5bd692 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "fa040004-d4e5-4f60-8a04-000000000004", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fb040004-e5f6-4071-9b04-000000000004\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"FeedDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T14:26:51.140Z\",\r\n \"EndTime\": \"2026-06-16T14:26:53.300Z\",\r\n \"CreationTime\": \"2026-06-16T14:26:50.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb040004-e5f6-4071-9b04-000000000004", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T14:26:53.150Z\",\r\n \"Message\": \"[Download File from URL] Don't know about such a host. (feeds.partner-portal.example:443)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T14:26:53.220Z\",\r\n \"Message\": \"Download File from URL: Don't know about such a host. (feeds.partner-portal.example:443)\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "fb040004-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"fb040004-e5f6-4071-9b04-000000000004\",\r\n \"StartTime\": \"2026-06-16T14:26:51.140Z\",\r\n \"EndTime\": \"2026-06-16T14:26:53.300Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"fc040004-a7b8-4192-9c04-000000000004\",\r\n \"Info\": \"Download File from URL: Don't know about such a host. (feeds.partner-portal.example:443)\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Http.HttpRequestException: Don't know about such a host. (feeds.partner-portal.example:443) ---> System.Net.Sockets.SocketException: Don't know about such a host.\",\r\n \"CreationTime\": \"2026-06-16T14:26:50.700Z\",\r\n \"ReleaseName\": \"FeedDownloader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7016004,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"fa040004-d4e5-4f60-8a04-000000000004\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T14:26:53.300Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"fb040004-e5f6-4071-9b04-000000000004\",\r\n \"Id\": 716010404\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb040004-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T14:26:51.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T14:26:52.180Z\",\r\n \"Message\": \"[Download File from URL] Resolving https://feeds.partner-portal.example/data/export.csv\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T14:26:53.150Z\",\r\n \"Message\": \"[Download File from URL] Don't know about such a host. (feeds.partner-portal.example:443)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T14:26:53.220Z\",\r\n \"Message\": \"Download File from URL: Don't know about such a host. (feeds.partner-portal.example:443)\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T14:26:53.270Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fa040004-d4e5-4f60-8a04-000000000004\",\r\n \"Name\": \"External Feeds\",\r\n \"Path\": \"External Feeds\",\r\n \"Description\": \"Production folder for external feed-download processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/task.yaml index 3bf27b9059..0220a13fc0 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-host-not-found/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/README.md new file mode 100644 index 0000000000..b7147803ba --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/README.md @@ -0,0 +1,42 @@ +# Download File from URL Failure - HTTP Client Reused in a Loop + +This scenario reproduces a `Download File from URL` failure inside a `For Each` +loop. The activity's internal HTTP client is reused across iterations, so the +first file downloads but a later iteration faults with `This instance has already +started one or more requests. Properties can only be modified before sending the +first request` (`System.InvalidOperationException`). + +## What this scenario uncovers + +**Root Cause:** Reused/improperly-disposed HTTP client across loop iterations — +the second iteration can't start a fresh request on a client that already sent +one. + +This maps to: +`references/activity-packages/file-operations/playbooks/download-file-httpclient-reused-in-loop.md` + +"First item works, later items fail" is the tell. The fix is a workflow change +(new `HttpClient` per iteration, or `GC.Collect`/`WaitForPendingFinalizers` after +the download) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `Download File from URL` inside a `For Each` over a list of URLs | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the reused HTTP client + across loop iterations and recommends a fresh `System.Net.Http.HttpClient` per + iteration (or `GC.Collect`/`WaitForPendingFinalizers` after the download), + without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/README.md deleted file mode 100644 index ecd9e2e309..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Download File from URL Failure - HTTP Client Reused in a Loop - -This scenario reproduces a `Download File from URL` failure inside a `For Each` -loop. The activity's internal HTTP client is reused across iterations, so the -first file downloads but a later iteration faults with `This instance has already -started one or more requests. Properties can only be modified before sending the -first request` (`System.InvalidOperationException`). - -## What this scenario uncovers - -**Root Cause:** Reused/improperly-disposed HTTP client across loop iterations — -the second iteration can't start a fresh request on a client that already sent -one. - -This maps to: -`references/activity-packages/file-operations/playbooks/download-file-httpclient-reused-in-loop.md` - -"First item works, later items fail" is the tell. The fix is a workflow change -(new `HttpClient` per iteration, or `GC.Collect`/`WaitForPendingFinalizers` after -the download) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `Download File from URL` inside a `For Each` over a list of URLs | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; logs show iteration 1 succeeds, iteration 2 faults | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the reused HTTP client - across loop iterations and recommends a fresh `System.Net.Http.HttpClient` per - iteration (or `GC.Collect`/`WaitForPendingFinalizers` after the download), - without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/manifest.json deleted file mode 100644 index 87b7e5f5b1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"fa010001-d4e5-4f60-8a01-000000000001\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa010001-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key fa010001-d4e5-4f60-8a01-000000000001 --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa010001-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"fb010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-get-fb010001-output-json.json" - }, - { - "match": "or jobs get fb010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-get-fb010001-output-json.json" - }, - { - "match": "or jobs logs \"fb010001-e5f6-4071-9b01-000000000001\" --level Error --output json", - "file": "or-jobs-logs-fb010001-level-error-output-json.json" - }, - { - "match": "or jobs logs fb010001-e5f6-4071-9b01-000000000001 --level Error --output json", - "file": "or-jobs-logs-fb010001-level-error-output-json.json" - }, - { - "match": "or jobs logs \"fb010001-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-logs-fb010001-output-json.json" - }, - { - "match": "or jobs logs fb010001-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-logs-fb010001-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 9332e9ab9d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "fa010001-d4e5-4f60-8a01-000000000001", - "Name": "Batch Downloads", - "Path": "Batch Downloads", - "Description": "Production folder for bulk file-download processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-get-fb010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-get-fb010001-output-json.json deleted file mode 100644 index 14abd161b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-get-fb010001-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "fb010001-e5f6-4071-9b01-000000000001", - "StartTime": "2026-06-16T08:14:30.140Z", - "EndTime": "2026-06-16T08:14:35.620Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "fc010001-a7b8-4192-9c01-000000000001", - "Info": "Download File from URL: This instance has already started one or more requests. Properties can only be modified before sending the first request.\r\n\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at ForEach \"For Each url\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.", - "CreationTime": "2026-06-16T08:14:29.700Z", - "ReleaseName": "BulkDownloader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7016001, - "OrganizationUnitId": 7791618, - "FolderKey": "fa010001-d4e5-4f60-8a01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T08:14:35.620Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "fb010001-e5f6-4071-9b01-000000000001", - "Id": 716010101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-list-folder-key-fa010001-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-list-folder-key-fa010001-state-faulted-output-json.json deleted file mode 100644 index 11764ca17f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-list-folder-key-fa010001-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "fb010001-e5f6-4071-9b01-000000000001", - "State": "Faulted", - "ProcessName": "BulkDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T08:14:30.140Z", - "EndTime": "2026-06-16T08:14:35.620Z", - "CreationTime": "2026-06-16T08:14:29.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-logs-fb010001-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-logs-fb010001-level-error-output-json.json deleted file mode 100644 index f69bbb712d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-logs-fb010001-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T08:14:35.470Z", - "Message": "[Download File from URL] This instance has already started one or more requests. Properties can only be modified before sending the first request." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T08:14:35.540Z", - "Message": "Download File from URL: This instance has already started one or more requests. Properties can only be modified before sending the first request.\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at ForEach \"For Each url\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-logs-fb010001-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-logs-fb010001-output-json.json deleted file mode 100644 index 8dfa200061..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/m/r/or-jobs-logs-fb010001-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T08:14:30.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T08:14:32.260Z", - "Message": "[Download File from URL] Downloaded report-1.pdf (iteration 1 of 3)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T08:14:35.470Z", - "Message": "[Download File from URL] This instance has already started one or more requests. Properties can only be modified before sending the first request. (iteration 2 of 3)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T08:14:35.540Z", - "Message": "Download File from URL: This instance has already started one or more requests. Properties can only be modified before sending the first request.\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at ForEach \"For Each url\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T08:14:35.600Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/uip-fixture.json new file mode 100644 index 0000000000..9111720ebd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "fa010001-d4e5-4f60-8a01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fb010001-e5f6-4071-9b01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BulkDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T08:14:30.140Z\",\r\n \"EndTime\": \"2026-06-16T08:14:35.620Z\",\r\n \"CreationTime\": \"2026-06-16T08:14:29.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb010001-e5f6-4071-9b01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T08:14:35.470Z\",\r\n \"Message\": \"[Download File from URL] This instance has already started one or more requests. Properties can only be modified before sending the first request.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T08:14:35.540Z\",\r\n \"Message\": \"Download File from URL: This instance has already started one or more requests. Properties can only be modified before sending the first request.\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at ForEach \\\"For Each url\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "fb010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"fb010001-e5f6-4071-9b01-000000000001\",\r\n \"StartTime\": \"2026-06-16T08:14:30.140Z\",\r\n \"EndTime\": \"2026-06-16T08:14:35.620Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"fc010001-a7b8-4192-9c01-000000000001\",\r\n \"Info\": \"Download File from URL: This instance has already started one or more requests. Properties can only be modified before sending the first request.\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at ForEach \\\"For Each url\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidOperationException: This instance has already started one or more requests. Properties can only be modified before sending the first request.\",\r\n \"CreationTime\": \"2026-06-16T08:14:29.700Z\",\r\n \"ReleaseName\": \"BulkDownloader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7016001,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"fa010001-d4e5-4f60-8a01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T08:14:35.620Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"fb010001-e5f6-4071-9b01-000000000001\",\r\n \"Id\": 716010101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb010001-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T08:14:30.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T08:14:32.260Z\",\r\n \"Message\": \"[Download File from URL] Downloaded report-1.pdf (iteration 1 of 3)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T08:14:35.470Z\",\r\n \"Message\": \"[Download File from URL] This instance has already started one or more requests. Properties can only be modified before sending the first request. (iteration 2 of 3)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T08:14:35.540Z\",\r\n \"Message\": \"Download File from URL: This instance has already started one or more requests. Properties can only be modified before sending the first request.\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at ForEach \\\"For Each url\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T08:14:35.600Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fa010001-d4e5-4f60-8a01-000000000001\",\r\n \"Name\": \"Batch Downloads\",\r\n \"Path\": \"Batch Downloads\",\r\n \"Description\": \"Production folder for bulk file-download processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/task.yaml index e6060aa2fc..7ee60ae182 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-httpclient-reused-in-loop/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/README.md new file mode 100644 index 0000000000..032bb61b99 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/README.md @@ -0,0 +1,41 @@ +# Download File from URL Failure - File Stuck as .tmp (Finalize Race) + +This scenario reproduces a `Download File from URL` finalize race. The download +"completes," but the next step (`Read CSV` on `data\daily.csv`) faults with +`Could not find file` because the file is still the temporary `daily.csv.tmp` — +the workflow consumed it before the stream finalized to the real name. + +## What this scenario uncovers + +**Root Cause:** The downstream `Read CSV` runs before the downloaded file is +finalized from `*.tmp` to `daily.csv`, so the target path doesn't resolve yet. +Intermittent (sometimes works). + +This maps to: +`references/activity-packages/file-operations/playbooks/download-file-stuck-tmp.md` + +The discriminator vs a plain file-not-found: a `daily.csv.tmp` is present and the +failure is timing-dependent. The fix is a workflow change (Retry Scope / File +Exists on the final name) — not a host action. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: `Download File from URL` (FileName `daily.csv`) followed by `Read CSV` on `data\daily.csv` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's diagnosis matches `RESOLUTION.md`: identifies the download/finalize race + (file still `.tmp` when the downstream step runs; intermittent) and recommends a + Retry Scope / File Exists gate on the **final** target extension before + consuming the file, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/README.md deleted file mode 100644 index 73cf07c52a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# Download File from URL Failure - File Stuck as .tmp (Finalize Race) - -This scenario reproduces a `Download File from URL` finalize race. The download -"completes," but the next step (`Read CSV` on `data\daily.csv`) faults with -`Could not find file` because the file is still the temporary `daily.csv.tmp` — -the workflow consumed it before the stream finalized to the real name. - -## What this scenario uncovers - -**Root Cause:** The downstream `Read CSV` runs before the downloaded file is -finalized from `*.tmp` to `daily.csv`, so the target path doesn't resolve yet. -Intermittent (sometimes works). - -This maps to: -`references/activity-packages/file-operations/playbooks/download-file-stuck-tmp.md` - -The discriminator vs a plain file-not-found: a `daily.csv.tmp` is present and the -failure is timing-dependent. The fix is a workflow change (Retry Scope / File -Exists on the final name) — not a host action. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: `Download File from URL` (FileName `daily.csv`) followed by `Read CSV` on `data\daily.csv` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; logs show the download completed and a `daily.csv.tmp` present when the read runs | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's diagnosis matches `RESOLUTION.md`: identifies the download/finalize race - (file still `.tmp` when the downstream step runs; intermittent) and recommends a - Retry Scope / File Exists gate on the **final** target extension before - consuming the file, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/manifest.json deleted file mode 100644 index 3ec20c5974..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"fa030003-d4e5-4f60-8a03-000000000003\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa030003-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key fa030003-d4e5-4f60-8a03-000000000003 --state Faulted --output json", - "file": "or-jobs-list-folder-key-fa030003-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"fb030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-get-fb030003-output-json.json" - }, - { - "match": "or jobs get fb030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-get-fb030003-output-json.json" - }, - { - "match": "or jobs logs \"fb030003-e5f6-4071-9b03-000000000003\" --level Error --output json", - "file": "or-jobs-logs-fb030003-level-error-output-json.json" - }, - { - "match": "or jobs logs fb030003-e5f6-4071-9b03-000000000003 --level Error --output json", - "file": "or-jobs-logs-fb030003-level-error-output-json.json" - }, - { - "match": "or jobs logs \"fb030003-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-logs-fb030003-output-json.json" - }, - { - "match": "or jobs logs fb030003-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-logs-fb030003-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 753f5a2efc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "fa030003-d4e5-4f60-8a03-000000000003", - "Name": "Report Intake", - "Path": "Report Intake", - "Description": "Production folder for download-and-read report processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-get-fb030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-get-fb030003-output-json.json deleted file mode 100644 index 0c01020246..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-get-fb030003-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "fb030003-e5f6-4071-9b03-000000000003", - "StartTime": "2026-06-16T11:40:08.140Z", - "EndTime": "2026-06-16T11:40:11.020Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "fc030003-a7b8-4192-9c03-000000000003", - "Info": "Read CSV: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\ReportDownloader\\data\\daily.csv'.\r\n\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileNotFoundException: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\ReportDownloader\\data\\daily.csv'. A partially-downloaded 'data\\daily.csv.tmp' is present.", - "CreationTime": "2026-06-16T11:40:07.700Z", - "ReleaseName": "ReportDownloader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7016003, - "OrganizationUnitId": 7791618, - "FolderKey": "fa030003-d4e5-4f60-8a03-000000000003", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T11:40:11.020Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "fb030003-e5f6-4071-9b03-000000000003", - "Id": 716010303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-list-folder-key-fa030003-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-list-folder-key-fa030003-state-faulted-output-json.json deleted file mode 100644 index 591619f143..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-list-folder-key-fa030003-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "fb030003-e5f6-4071-9b03-000000000003", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T11:40:08.140Z", - "EndTime": "2026-06-16T11:40:11.020Z", - "CreationTime": "2026-06-16T11:40:07.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-logs-fb030003-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-logs-fb030003-level-error-output-json.json deleted file mode 100644 index a4a67ceb7e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-logs-fb030003-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T11:40:10.870Z", - "Message": "[Read CSV] Could not find file 'C:\\ProgramData\\UiPath\\Packages\\ReportDownloader\\data\\daily.csv'. A partially-downloaded 'data\\daily.csv.tmp' is present." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T11:40:10.940Z", - "Message": "Read CSV: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\ReportDownloader\\data\\daily.csv'.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-logs-fb030003-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-logs-fb030003-output-json.json deleted file mode 100644 index 812c8d734c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/m/r/or-jobs-logs-fb030003-output-json.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T11:40:08.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T11:40:09.760Z", - "Message": "[Download File from URL] Completed download from https://files.example.com/exports/daily.csv" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T11:40:10.210Z", - "Message": "[Read CSV] Reading data\\daily.csv (Delimiter=Comma, AddHeaders=True); folder currently contains data\\daily.csv.tmp" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T11:40:10.870Z", - "Message": "[Read CSV] Could not find file 'C:\\ProgramData\\UiPath\\Packages\\ReportDownloader\\data\\daily.csv'. A partially-downloaded 'data\\daily.csv.tmp' is present." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T11:40:10.940Z", - "Message": "Read CSV: Could not find file 'C:\\ProgramData\\UiPath\\Packages\\ReportDownloader\\data\\daily.csv'.\r\nin Main.xaml\r\n at ReadCsvFile \"Read CSV\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T11:40:10.990Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/uip-fixture.json new file mode 100644 index 0000000000..86c0e54752 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "fa030003-d4e5-4f60-8a03-000000000003", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fb030003-e5f6-4071-9b03-000000000003\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T11:40:08.140Z\",\r\n \"EndTime\": \"2026-06-16T11:40:11.020Z\",\r\n \"CreationTime\": \"2026-06-16T11:40:07.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb030003-e5f6-4071-9b03-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T11:40:10.870Z\",\r\n \"Message\": \"[Read CSV] Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\ReportDownloader\\\\data\\\\daily.csv'. A partially-downloaded 'data\\\\daily.csv.tmp' is present.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T11:40:10.940Z\",\r\n \"Message\": \"Read CSV: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\ReportDownloader\\\\data\\\\daily.csv'.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "fb030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"fb030003-e5f6-4071-9b03-000000000003\",\r\n \"StartTime\": \"2026-06-16T11:40:08.140Z\",\r\n \"EndTime\": \"2026-06-16T11:40:11.020Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"fc030003-a7b8-4192-9c03-000000000003\",\r\n \"Info\": \"Read CSV: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\ReportDownloader\\\\data\\\\daily.csv'.\\r\\n\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\ReportDownloader\\\\data\\\\daily.csv'. A partially-downloaded 'data\\\\daily.csv.tmp' is present.\",\r\n \"CreationTime\": \"2026-06-16T11:40:07.700Z\",\r\n \"ReleaseName\": \"ReportDownloader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7016003,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"fa030003-d4e5-4f60-8a03-000000000003\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T11:40:11.020Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"fb030003-e5f6-4071-9b03-000000000003\",\r\n \"Id\": 716010303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "fb030003-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T11:40:08.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T11:40:09.760Z\",\r\n \"Message\": \"[Download File from URL] Completed download from https://files.example.com/exports/daily.csv\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T11:40:10.210Z\",\r\n \"Message\": \"[Read CSV] Reading data\\\\daily.csv (Delimiter=Comma, AddHeaders=True); folder currently contains data\\\\daily.csv.tmp\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T11:40:10.870Z\",\r\n \"Message\": \"[Read CSV] Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\ReportDownloader\\\\data\\\\daily.csv'. A partially-downloaded 'data\\\\daily.csv.tmp' is present.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T11:40:10.940Z\",\r\n \"Message\": \"Read CSV: Could not find file 'C:\\\\ProgramData\\\\UiPath\\\\Packages\\\\ReportDownloader\\\\data\\\\daily.csv'.\\r\\nin Main.xaml\\r\\n at ReadCsvFile \\\"Read CSV\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T11:40:10.990Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fa030003-d4e5-4f60-8a03-000000000003\",\r\n \"Name\": \"Report Intake\",\r\n \"Path\": \"Report Intake\",\r\n \"Description\": \"Production folder for download-and-read report processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/task.yaml index ae24ffbe02..2f47eacdbf 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/download-file-stuck-tmp/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/README.md new file mode 100644 index 0000000000..d8dff580d5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/README.md @@ -0,0 +1,30 @@ +# Digitize Document — invalid DU OCR API key (read the server response, not the AggregateException wrapper) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the invalid-API-key failure of `UiPath.IntelligentOCR.Activities` Digitize Document (UiPath Document OCR engine). + +## What this exercises + +A `Digitize Document` (UiPath Document OCR engine) calls the Document Understanding OCR service, which rejects the request because the API key is invalid. The job ends Faulted with `System.AggregateException` whose inner message is `Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey` (from `UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr`). The agent must read the **inner server response** as the cause (invalid UiPath Document OCR `ApiKey`) — not stop at the generic `One or more errors occurred.` `AggregateException` wrapper, and not blame the document content, a missing/corrupt file, the **Digitizer PDF-component license** (`Invalid license for the PDF component`, a different local failure), DU-not-enabled-on-tenant, page-units, or request-size. The fix is to set a valid DU OCR API key (from the tenant's AI Units). + +Signature captured **verbatim from a live `uip rpa` run** of Digitize Document + UiPath Document OCR against an invalid `ApiKey`; workflow/process names neutralized, no real key or document. Maps to the [du-license-or-endpoint-rejected](../../../../../skills/uipath-troubleshoot/references/activity-packages/intelligent-ocr-activities/playbooks/du-license-or-endpoint-rejected.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, AggregateException + `Invalid API key specified` / `UiPathOCRInvalidApiKey`) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the server response is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/1bd6291f11.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/1bd6291f11.json deleted file mode 100644 index a4d0c19c3e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/1bd6291f11.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "StartTime": "2026-06-20T13:47:20.330Z", - "EndTime": "2026-06-20T13:47:26.901Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "e6f7a8b9-c0d1-4e2f-3a4b-5c6d7e8f9a0b", - "Info": "Source: Digitize Document\r\nMessage: One or more errors occurred. (Request CorrelationId: 11fe739d-f26f-4631-9a47-8ad14cee388a\nRequest PredictionId: dxESNP+Cfd/lURIXfBaFd6eHzUvHTZKA7mdpXTRI848=_39ca2d10-d072-4747-b92a-efbe827a1a92\nServer response: Invalid API key specified Error:UiPathOCRInvalidApiKey CF-RAY: cid-v1:354c7cb9-ae5a-4d16-84a7-f13242bbac6d. AppId: )\r\nException Type: System.AggregateException\r\nin Main.xaml\r\n at DigitizeDocument \"Digitize Document\"\r\n at Sequence \"Main\"\r\n at Main \"Main\"\r\n\r\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr(CachingPageRenderer renderer, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CancellationToken token)\r\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ProcessPageWithOcr(Content page, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CachingPageRenderer renderer, CancellationToken token)\r\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.DocumentDigitizer.Digitize(Content content, DigitizationSettings settings, IOcrEngine ocrEngine, CancellationToken token)\r\n at UiPath.IntelligentOCR.Digitization.IntelligentOcrDigitizer.Digitize(Content content, IOcrEngine ocrEngine, ApplyOcrOnPdf applyOcrOnPdf, Boolean detectCheckboxes, IDigitizationScheduler scheduler, IDigitizerTelemetryService telemetryService, CancellationToken token)\r\n at UiPath.IntelligentOCR.Activities.Digitization.DigitizeDocument.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)", - "CreationTime": "2026-06-20T13:47:19.100Z", - "ReleaseName": "DocIntake", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1109033, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T13:47:26.901Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c5f2a8d13e6b4f9c8a2d1b3c4d5e6f70", - "Id": 50620044 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/275c99e309.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/275c99e309.json deleted file mode 100644 index 6da112bef5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/275c99e309.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T13:47:26.500Z", - "Message": "Digitize Document: One or more errors occurred. (Request CorrelationId: 11fe739d-f26f-4631-9a47-8ad14cee388a\nRequest PredictionId: dxESNP+Cfd/lURIXfBaFd6eHzUvHTZKA7mdpXTRI848=_39ca2d10-d072-4747-b92a-efbe827a1a92\nServer response: Invalid API key specified Error:UiPathOCRInvalidApiKey CF-RAY: cid-v1:354c7cb9-ae5a-4d16-84a7-f13242bbac6d. AppId: )\r\nException Type: System.AggregateException\r\nin Main.xaml\r\n at DigitizeDocument \"Digitize Document\"\r\n at Sequence \"Main\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.AggregateException: One or more errors occurred. (Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey)\r\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr(CachingPageRenderer renderer, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CancellationToken token)\r\n at UiPath.IntelligentOCR.Activities.Digitization.DigitizeDocument.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/README.md deleted file mode 100644 index e0eb141e7c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Digitize Document — invalid DU OCR API key (read the server response, not the AggregateException wrapper) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the invalid-API-key failure of `UiPath.IntelligentOCR.Activities` Digitize Document (UiPath Document OCR engine). - -## What this exercises - -A `Digitize Document` (UiPath Document OCR engine) calls the Document Understanding OCR service, which rejects the request because the API key is invalid. The job ends Faulted with `System.AggregateException` whose inner message is `Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey` (from `UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr`). The agent must read the **inner server response** as the cause (invalid UiPath Document OCR `ApiKey`) — not stop at the generic `One or more errors occurred.` `AggregateException` wrapper, and not blame the document content, a missing/corrupt file, the **Digitizer PDF-component license** (`Invalid license for the PDF component`, a different local failure), DU-not-enabled-on-tenant, page-units, or request-size. The fix is to set a valid DU OCR API key (from the tenant's AI Units). - -Signature captured **verbatim from a live `uip rpa` run** of Digitize Document + UiPath Document OCR against an invalid `ApiKey`; workflow/process names neutralized, no real key or document. Maps to the [du-license-or-endpoint-rejected](../../../../../skills/uipath-troubleshoot/references/activity-packages/intelligent-ocr-activities/playbooks/du-license-or-endpoint-rejected.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, AggregateException + `Invalid API key specified` / `UiPathOCRInvalidApiKey`) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the server response is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/a2a635bd85.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/a2a635bd85.json deleted file mode 100644 index 402e5cd23c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/a2a635bd85.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "State": "Faulted", - "StartTime": "2026-06-20T13:47:20.330Z", - "EndTime": "2026-06-20T13:47:26.901Z", - "ReleaseName": "DocIntake", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50620044 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/manifest.json deleted file mode 100644 index 9b7c4ba397..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.IntelligentOCR.Activities Digitize Document (UiPath Document OCR engine) faults with System.AggregateException whose inner message is 'Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey' (from PageDigitizer.ApplyOcr) because the UiPath Document OCR API key is invalid. Signature captured verbatim from a live uip rpa run; workflow/process names neutralized, no real key/document. The agent must read the inner server response as the cause (invalid OCR ApiKey), not stop at the AggregateException wrapper, and not blame the document, a missing file, the Digitizer PDF-component license, or tenant-not-enabled. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "a2a635bd85.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "a2a635bd85.json" - }, - { - "match": "or jobs list", - "file": "a2a635bd85.json" - }, - { - "match": "or jobs get c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "file": "1bd6291f11.json" - }, - { - "match": "or jobs logs c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70 --level Error", - "file": "275c99e309.json" - }, - { - "match": "or jobs logs c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "file": "275c99e309.json" - }, - { - "match": "or jobs traces c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", - "min": 1, - "description": "Agent fetches the faulted job to read the 'Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey' signature (System.AggregateException) from the Info / Error log." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/uip-fixture.json new file mode 100644 index 0000000000..f68a26e95d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T13:47:20.330Z\",\r\n \"EndTime\": \"2026-06-20T13:47:26.901Z\",\r\n \"ReleaseName\": \"DocIntake\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50620044\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T13:47:26.500Z\",\r\n \"Message\": \"Digitize Document: One or more errors occurred. (Request CorrelationId: 11fe739d-f26f-4631-9a47-8ad14cee388a\\nRequest PredictionId: dxESNP+Cfd/lURIXfBaFd6eHzUvHTZKA7mdpXTRI848=_39ca2d10-d072-4747-b92a-efbe827a1a92\\nServer response: Invalid API key specified Error:UiPathOCRInvalidApiKey CF-RAY: cid-v1:354c7cb9-ae5a-4d16-84a7-f13242bbac6d. AppId: )\\r\\nException Type: System.AggregateException\\r\\nin Main.xaml\\r\\n at DigitizeDocument \\\"Digitize Document\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException: One or more errors occurred. (Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey)\\r\\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr(CachingPageRenderer renderer, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CancellationToken token)\\r\\n at UiPath.IntelligentOCR.Activities.Digitization.DigitizeDocument.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T13:47:20.330Z\",\r\n \"EndTime\": \"2026-06-20T13:47:26.901Z\",\r\n \"ReleaseName\": \"DocIntake\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50620044\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70\",\r\n \"StartTime\": \"2026-06-20T13:47:20.330Z\",\r\n \"EndTime\": \"2026-06-20T13:47:26.901Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"e6f7a8b9-c0d1-4e2f-3a4b-5c6d7e8f9a0b\",\r\n \"Info\": \"Source: Digitize Document\\r\\nMessage: One or more errors occurred. (Request CorrelationId: 11fe739d-f26f-4631-9a47-8ad14cee388a\\nRequest PredictionId: dxESNP+Cfd/lURIXfBaFd6eHzUvHTZKA7mdpXTRI848=_39ca2d10-d072-4747-b92a-efbe827a1a92\\nServer response: Invalid API key specified Error:UiPathOCRInvalidApiKey CF-RAY: cid-v1:354c7cb9-ae5a-4d16-84a7-f13242bbac6d. AppId: )\\r\\nException Type: System.AggregateException\\r\\nin Main.xaml\\r\\n at DigitizeDocument \\\"Digitize Document\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr(CachingPageRenderer renderer, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CancellationToken token)\\r\\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ProcessPageWithOcr(Content page, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CachingPageRenderer renderer, CancellationToken token)\\r\\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.DocumentDigitizer.Digitize(Content content, DigitizationSettings settings, IOcrEngine ocrEngine, CancellationToken token)\\r\\n at UiPath.IntelligentOCR.Digitization.IntelligentOcrDigitizer.Digitize(Content content, IOcrEngine ocrEngine, ApplyOcrOnPdf applyOcrOnPdf, Boolean detectCheckboxes, IDigitizationScheduler scheduler, IDigitizerTelemetryService telemetryService, CancellationToken token)\\r\\n at UiPath.IntelligentOCR.Activities.Digitization.DigitizeDocument.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\",\r\n \"CreationTime\": \"2026-06-20T13:47:19.100Z\",\r\n \"ReleaseName\": \"DocIntake\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1109033,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T13:47:26.901Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c5f2a8d13e6b4f9c8a2d1b3c4d5e6f70\",\r\n \"Id\": 50620044\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T13:47:26.500Z\",\r\n \"Message\": \"Digitize Document: One or more errors occurred. (Request CorrelationId: 11fe739d-f26f-4631-9a47-8ad14cee388a\\nRequest PredictionId: dxESNP+Cfd/lURIXfBaFd6eHzUvHTZKA7mdpXTRI848=_39ca2d10-d072-4747-b92a-efbe827a1a92\\nServer response: Invalid API key specified Error:UiPathOCRInvalidApiKey CF-RAY: cid-v1:354c7cb9-ae5a-4d16-84a7-f13242bbac6d. AppId: )\\r\\nException Type: System.AggregateException\\r\\nin Main.xaml\\r\\n at DigitizeDocument \\\"Digitize Document\\\"\\r\\n at Sequence \\\"Main\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException: One or more errors occurred. (Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey)\\r\\n at UiPath.DocumentUnderstanding.Digitizer.Digitization.PageDigitizer.ApplyOcr(CachingPageRenderer renderer, IOcrEngine ocrEngine, DigitizationSettings settings, String contentId, CancellationToken token)\\r\\n at UiPath.IntelligentOCR.Activities.Digitization.DigitizeDocument.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T13:47:20.330Z\",\r\n \"EndTime\": \"2026-06-20T13:47:26.901Z\",\r\n \"ReleaseName\": \"DocIntake\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50620044\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get c5f2a8d1-3e6b-4f9c-8a2d-1b3c4d5e6f70", + "min": 1, + "description": "Agent fetches the faulted job to read the 'Server response: Invalid API key specified Error:UiPathOCRInvalidApiKey' signature (System.AggregateException) from the Info / Error log." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/task.yaml index 3f3b5e2e8a..350b036cb4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/du-digitize-license-not-validated/task.yaml @@ -24,20 +24,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/README.md new file mode 100644 index 0000000000..e92fd24edb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/README.md @@ -0,0 +1,58 @@ +# Excel Append Range — Column Schema Mismatch (By-Position Trap) + +This scenario reproduces an Append Range failure where the source +`DataTable`'s 4 columns and the target sheet's 4-column header row +have the same NAMES but in different ORDERS. Append Range writes +by position (not by header name), so a numeric column lands in a +text column and Excel COM rejects the cell. The job ends with: + +``` +System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error. +``` + +## What this scenario uncovers + +**Root Cause:** The source DataTable's column order is +`[EmployeeId, HireDate, FullName, Department]` but the target +sheet's existing header is `[FullName, EmployeeId, Department, HireDate]`. +Append Range writes the source's column 0 (numeric `EmployeeId`) +into the target's column A (text `FullName`) — Excel COM rejects +the cell with `0x800A03EC`. The workflow author likely assumed +Append Range would map columns by header name; it doesn't. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/append-range-failures.md` +(the "Column schema mismatch" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelHRProcess` project — `Invoke FetchNewHires.xaml` → `Use Excel File` → `Append Range` with no intermediate column-reorder transformation | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (COMException 0x800A03EC +on AppendRange_1, stack through `Range.set_Value`) → `jobs logs` +(the source vs. target column-order Trace + the cell-rejection +explanation are the smoking gun) → workflow source review (confirms +no column-reorder step between the sub-workflow output and the +Append Range) → conclude branch 5. + +> **Note on the by-position semantics.** This is the playbook's +> counterintuitive trap: visual presence of column names in both +> ends creates a false expectation of by-name mapping. The test +> grades agents that recognize Append Range's by-position semantics +> as the cause AND recommend an explicit column-reorder step +> (DataTable.DefaultView.ToTable with the target's column-name +> list). + +> **Note on fixtures.** Synthetic. The HR sub-workflow, column +> name choices, and exact COMException stack frame are +> placeholders. The test grades whether the agent identifies the +> position-vs-name mapping trap AND recommends a viable structural +> fix (reorder source, restructure target, or migrate to Write +> Range with explicit mapping). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/019eb08ebf.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/019eb08ebf.json deleted file mode 100644 index d8858eb51b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/019eb08ebf.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ff2222cc-001", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-28T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-002", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-28T08:00:01.418Z", - "Message": "[ExcelHRProcess] Appending this week's new hires to C:\\Robot\\Data\\employees.xlsx", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-003", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-28T08:00:02.010Z", - "Message": "Invoke: FetchNewHires.xaml — returned dtNewHires with 12 rows, 4 columns (EmployeeId, HireDate, FullName, Department)", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-004", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-28T08:00:02.310Z", - "Message": "Use Excel File: employees.xlsx — workbook opened (COM provider)", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-005", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-28T08:00:02.510Z", - "Message": "Append Range: new hires — target sheet 'Employees' header row read: ['FullName', 'EmployeeId', 'Department', 'HireDate'] (4 columns)", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-006", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-28T08:00:02.710Z", - "Message": "Append Range: new hires — last data row in target sheet 'Employees': row 847. Append target starts at row 848.", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-007", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-28T08:00:03.110Z", - "Message": "Append Range: new hires — marshalling row 1 of 12 to Excel COM (A848:D848): writing source column 0 'EmployeeId' (value=10847, type=Int64) to target column A 'FullName' (stored type: Text)", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-008", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-28T08:00:03.510Z", - "Message": "Append Range: new hires — COM Range.set_Value rejected cell A848: source column order ['EmployeeId', 'HireDate', 'FullName', 'Department'] vs target header order ['FullName', 'EmployeeId', 'Department', 'HireDate'] — Append writes by POSITION, columns don't align", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff2222cc-009", - "JobKey": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-28T08:00:03.802Z", - "Message": "Append Range: new hires — System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.", - "ProcessName": "ExcelHRProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/1d4a124ef4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/1d4a124ef4.json deleted file mode 100644 index 070ac1d1af..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/1d4a124ef4.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0099999-3333-4444-5555-666677778888", - "DisplayName": "HROps", - "FullyQualifiedName": "HROps", - "Description": "Folder for weekly HR new-hire feed automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6090001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/2d254f8714.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/2d254f8714.json deleted file mode 100644 index fd6b9aa044..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/2d254f8714.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-28T08:00:01.300Z", - "EndTime": "2026-05-28T08:00:03.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ff22222-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\r\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\r\n at Microsoft.Office.Interop.Excel.Range.set_Value(Object RangeValueDataType, Object value)\r\n at UiPath.Excel.Activities.Business.AppendRangeX.WriteRowBlock(Range target, DataRow row)\r\n at UiPath.Excel.Activities.Business.AppendRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-28T08:00:01.005Z", - "ReleaseName": "ExcelHRProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680901, - "OrganizationUnitId": 6090001, - "FolderKey": "f0099999-3333-4444-5555-666677778888", - "Reference": "WeeklyNewHiresA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-28T08:00:03.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "ActivityName": "AppendRange_1", - "ActivityDisplayName": "Append Range: new hires", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680999001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/88af12778b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/88af12778b.json deleted file mode 100644 index 8a8092c010..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/88af12778b.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-28T08:00:01.005Z"}, - {"Key": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-28T08:00:01.300Z"}, - {"Key": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-28T08:00:03.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/README.md deleted file mode 100644 index 91841e9469..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# Excel Append Range — Column Schema Mismatch (By-Position Trap) - -This scenario reproduces an Append Range failure where the source -`DataTable`'s 4 columns and the target sheet's 4-column header row -have the same NAMES but in different ORDERS. Append Range writes -by position (not by header name), so a numeric column lands in a -text column and Excel COM rejects the cell. The job ends with: - -``` -System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error. -``` - -## What this scenario uncovers - -**Root Cause:** The source DataTable's column order is -`[EmployeeId, HireDate, FullName, Department]` but the target -sheet's existing header is `[FullName, EmployeeId, Department, HireDate]`. -Append Range writes the source's column 0 (numeric `EmployeeId`) -into the target's column A (text `FullName`) — Excel COM rejects -the cell with `0x800A03EC`. The workflow author likely assumed -Append Range would map columns by header name; it doesn't. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/append-range-failures.md` -(the "Column schema mismatch" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelHRProcess` project — `Invoke FetchNewHires.xaml` → `Use Excel File` → `Append Range` with no intermediate column-reorder transformation | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture BOTH column orders side-by-side AND the cell-level COM rejection naming the type mismatch | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (COMException 0x800A03EC -on AppendRange_1, stack through `Range.set_Value`) → `jobs logs` -(the source vs. target column-order Trace + the cell-rejection -explanation are the smoking gun) → workflow source review (confirms -no column-reorder step between the sub-workflow output and the -Append Range) → conclude branch 5. - -> **Note on the by-position semantics.** This is the playbook's -> counterintuitive trap: visual presence of column names in both -> ends creates a false expectation of by-name mapping. The test -> grades agents that recognize Append Range's by-position semantics -> as the cause AND recommend an explicit column-reorder step -> (DataTable.DefaultView.ToTable with the target's column-name -> list). - -> **Note on fixtures.** Synthetic. The HR sub-workflow, column -> name choices, and exact COMException stack frame are -> placeholders. The test grades whether the agent identifies the -> position-vs-name mapping trap AND recommends a viable structural -> fix (reorder source, restructure target, or migrate to Write -> Range with explicit mapping). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/fb4f8a02fb.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/fb4f8a02fb.json deleted file mode 100644 index 2e21b1ec8b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/fb4f8a02fb.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ff222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-28T08:00:01.300Z", - "EndTime": "2026-05-28T08:00:03.812Z", - "State": "Faulted", - "ReleaseName": "ExcelHRProcess", - "Source": "Schedule", - "Reference": "WeeklyNewHiresA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0099999-3333-4444-5555-666677778888", - "Info": "System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/manifest.json deleted file mode 100644 index 08faa685cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Append Range new-hire append scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "1d4a124ef4.json"}, - {"match": "or folders list --output json", "file": "1d4a124ef4.json"}, - {"match": "or jobs list --folder-key \"f0099999-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "fb4f8a02fb.json"}, - {"match": "or jobs list --folder-key f0099999-3333-4444-5555-666677778888 --state Faulted --output json", "file": "fb4f8a02fb.json"}, - {"match": "or jobs get \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "2d254f8714.json"}, - {"match": "or jobs get ff222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "2d254f8714.json"}, - {"match": "or jobs history \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "88af12778b.json"}, - {"match": "or jobs history ff222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "88af12778b.json"}, - {"match": "or jobs logs \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "019eb08ebf.json"}, - {"match": "or jobs logs ff222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "019eb08ebf.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..9f6065bac7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0099999-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-28T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-28T08:00:03.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelHRProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"WeeklyNewHiresA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0099999-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ff222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-28T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-28T08:00:03.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ff22222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\\r\\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\\r\\n at Microsoft.Office.Interop.Excel.Range.set_Value(Object RangeValueDataType, Object value)\\r\\n at UiPath.Excel.Activities.Business.AppendRangeX.WriteRowBlock(Range target, DataRow row)\\r\\n at UiPath.Excel.Activities.Business.AppendRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-28T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelHRProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680901,\r\n \"OrganizationUnitId\": 6090001,\r\n \"FolderKey\": \"f0099999-3333-4444-5555-666677778888\",\r\n \"Reference\": \"WeeklyNewHiresA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-28T08:00:03.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"ActivityName\": \"AppendRange_1\",\r\n \"ActivityDisplayName\": \"Append Range: new hires\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680999001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ff222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-28T08:00:01.005Z\"},\r\n {\"Key\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-28T08:00:01.300Z\"},\r\n {\"Key\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-28T08:00:03.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ff222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ff2222cc-001\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-28T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-002\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-28T08:00:01.418Z\",\r\n \"Message\": \"[ExcelHRProcess] Appending this week's new hires to C:\\\\Robot\\\\Data\\\\employees.xlsx\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-003\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-28T08:00:02.010Z\",\r\n \"Message\": \"Invoke: FetchNewHires.xaml — returned dtNewHires with 12 rows, 4 columns (EmployeeId, HireDate, FullName, Department)\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-004\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-28T08:00:02.310Z\",\r\n \"Message\": \"Use Excel File: employees.xlsx — workbook opened (COM provider)\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-005\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-28T08:00:02.510Z\",\r\n \"Message\": \"Append Range: new hires — target sheet 'Employees' header row read: ['FullName', 'EmployeeId', 'Department', 'HireDate'] (4 columns)\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-006\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-28T08:00:02.710Z\",\r\n \"Message\": \"Append Range: new hires — last data row in target sheet 'Employees': row 847. Append target starts at row 848.\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-007\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-28T08:00:03.110Z\",\r\n \"Message\": \"Append Range: new hires — marshalling row 1 of 12 to Excel COM (A848:D848): writing source column 0 'EmployeeId' (value=10847, type=Int64) to target column A 'FullName' (stored type: Text)\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-008\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-28T08:00:03.510Z\",\r\n \"Message\": \"Append Range: new hires — COM Range.set_Value rejected cell A848: source column order ['EmployeeId', 'HireDate', 'FullName', 'Department'] vs target header order ['FullName', 'EmployeeId', 'Department', 'HireDate'] — Append writes by POSITION, columns don't align\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff2222cc-009\",\r\n \"JobKey\": \"ff222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-28T08:00:03.802Z\",\r\n \"Message\": \"Append Range: new hires — System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\",\r\n \"ProcessName\": \"ExcelHRProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0099999-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"HROps\",\r\n \"FullyQualifiedName\": \"HROps\",\r\n \"Description\": \"Folder for weekly HR new-hire feed automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6090001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/task.yaml index 836bfacb1a..9411d3e8ff 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-column-schema-mismatch/task.yaml @@ -28,21 +28,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/README.md new file mode 100644 index 0000000000..9b6ea3dbec --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/README.md @@ -0,0 +1,57 @@ +# Excel Append Range — Hidden Rows in Target Append Region (v2.8.5+) + +This scenario reproduces an Append Range failure where the target +sheet has an active AutoFilter hiding rows in the computed append +region. Package v2.8.5+ throws an explicit BusinessException to +prevent the silent data loss that pre-2.8.5 versions caused. The +job ends with: + +``` +UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089). +``` + +## What this scenario uncovers + +**Root Cause:** The target sheet `Transactions` has an active +AutoFilter on the `Status` column hiding rows 4012-4089 +(`Status='Reconciled'`). The Append Range activity's target-region +validator (v2.8.5+) detected the hidden rows in or near the +computed append target and refused to proceed. The workflow has no +filter-normalization step before the append. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/append-range-failures.md` +(the "Hidden rows in target append region" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelLedgerProcess` project — `Use Excel File` → `Append Range` against a sheet with an active AutoFilter, no filter-normalization step | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException naming +hidden rows in the append region) → `jobs logs` (AutoFilter +detection + v2.8.5+ guard activation + hidden row range +identification — all on consecutive Trace lines) → workflow source +review (confirms no Remove Data Filter before Append Range) → +conclude branch 6. + +> **Note on the anti-pattern trap.** The natural temptation is to +> downgrade the package to pre-2.8.5 to "silence the error." The +> playbook explicitly rejects this — pre-2.8.5 versions silently +> append over hidden rows, causing the data loss the explicit +> error was added to catch. The test penalizes agents that +> recommend a package downgrade and rewards those that recommend +> a filter-normalization step (Remove Data Filter or equivalent). + +> **Note on fixtures.** Synthetic. The transactions sub-workflow, +> exact hidden row range, and target-region computation are +> placeholders. The test grades whether the agent identifies the +> AutoFilter + v2.8.5+ guard combination as the root AND +> recommends a viable fix (remove the filter, route to a +> different sheet, or unhide explicitly). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/296981e65e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/296981e65e.json deleted file mode 100644 index b1b1f48ec3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/296981e65e.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ff3333cc-001", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-29T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-002", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-29T08:00:01.418Z", - "Message": "[ExcelLedgerProcess] Appending today's transactions to C:\\Robot\\Data\\ledger.xlsx", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-003", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-29T08:00:01.910Z", - "Message": "Invoke: FetchTransactions.xaml — returned dtTransactions with 78 rows, 6 columns (TxnId, TxnDate, Account, Amount, Currency, Status)", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-004", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-29T08:00:02.110Z", - "Message": "Use Excel File: ledger.xlsx — workbook opened (OpenXML provider); UiPath.Excel.Activities version 2.24.7 (>= 2.8.5)", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-005", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-29T08:00:02.310Z", - "Message": "Append Range: today's transactions — target sheet 'Transactions' inspection: AutoFilter active on column 'Status' (hiding rows where Status='Reconciled')", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-006", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-29T08:00:02.510Z", - "Message": "Append Range: today's transactions — last data row on sheet 'Transactions': row 4100. Computed append region: rows 4101-4178 (78 rows for the dtTransactions payload).", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-007", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-29T08:00:02.710Z", - "Message": "Append Range: today's transactions — target region validation (v2.8.5+): hidden rows detected in append target — sheet rows 4012-4089 are hidden by the AutoFilter and overlap with the computed append region's lookback. Pre-2.8.5 versions would have silently appended over the hidden region; v2.8.5+ throws to prevent data loss.", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff3333cc-008", - "JobKey": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-29T08:00:03.202Z", - "Message": "Append Range: today's transactions — UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089).", - "ProcessName": "ExcelLedgerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/84b50af139.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/84b50af139.json deleted file mode 100644 index 01242b2127..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/84b50af139.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-29T08:00:01.300Z", - "EndTime": "2026-05-29T08:00:03.212Z", - "State": "Faulted", - "ReleaseName": "ExcelLedgerProcess", - "Source": "Schedule", - "Reference": "DailyLedgerA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00aaaaa-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089).", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/README.md deleted file mode 100644 index 9b34f1b2a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Excel Append Range — Hidden Rows in Target Append Region (v2.8.5+) - -This scenario reproduces an Append Range failure where the target -sheet has an active AutoFilter hiding rows in the computed append -region. Package v2.8.5+ throws an explicit BusinessException to -prevent the silent data loss that pre-2.8.5 versions caused. The -job ends with: - -``` -UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089). -``` - -## What this scenario uncovers - -**Root Cause:** The target sheet `Transactions` has an active -AutoFilter on the `Status` column hiding rows 4012-4089 -(`Status='Reconciled'`). The Append Range activity's target-region -validator (v2.8.5+) detected the hidden rows in or near the -computed append target and refused to proceed. The workflow has no -filter-normalization step before the append. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/append-range-failures.md` -(the "Hidden rows in target append region" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelLedgerProcess` project — `Use Excel File` → `Append Range` against a sheet with an active AutoFilter, no filter-normalization step | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture (a) the AutoFilter detection on column 'Status', (b) the v2.8.5+ package-version note, (c) the target-region validation that explicitly identifies the hidden row range, and (d) the BusinessException | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException naming -hidden rows in the append region) → `jobs logs` (AutoFilter -detection + v2.8.5+ guard activation + hidden row range -identification — all on consecutive Trace lines) → workflow source -review (confirms no Remove Data Filter before Append Range) → -conclude branch 6. - -> **Note on the anti-pattern trap.** The natural temptation is to -> downgrade the package to pre-2.8.5 to "silence the error." The -> playbook explicitly rejects this — pre-2.8.5 versions silently -> append over hidden rows, causing the data loss the explicit -> error was added to catch. The test penalizes agents that -> recommend a package downgrade and rewards those that recommend -> a filter-normalization step (Remove Data Filter or equivalent). - -> **Note on fixtures.** Synthetic. The transactions sub-workflow, -> exact hidden row range, and target-region computation are -> placeholders. The test grades whether the agent identifies the -> AutoFilter + v2.8.5+ guard combination as the root AND -> recommends a viable fix (remove the filter, route to a -> different sheet, or unhide explicitly). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/c53ac0c2b8.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/c53ac0c2b8.json deleted file mode 100644 index 42591ef8b1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/c53ac0c2b8.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f00aaaaa-3333-4444-5555-666677778888", - "DisplayName": "LedgerOps", - "FullyQualifiedName": "LedgerOps", - "Description": "Folder for daily transaction-ledger append automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6100001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/d4b97a9106.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/d4b97a9106.json deleted file mode 100644 index 4165e621c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/d4b97a9106.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-29T08:00:01.005Z"}, - {"Key": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-29T08:00:01.300Z"}, - {"Key": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-29T08:00:03.212Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/ff43493cfb.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/ff43493cfb.json deleted file mode 100644 index 1d0895cd95..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/ff43493cfb.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-29T08:00:01.300Z", - "EndTime": "2026-05-29T08:00:03.212Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ff33333-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089).\r\n at UiPath.Excel.Activities.Business.AppendRangeX.ValidateTargetRegion(Worksheet sheet, Int32 startRow, Int32 endRow)\r\n at UiPath.Excel.Activities.Business.AppendRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-29T08:00:01.005Z", - "ReleaseName": "ExcelLedgerProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681001, - "OrganizationUnitId": 6100001, - "FolderKey": "f00aaaaa-3333-4444-5555-666677778888", - "Reference": "DailyLedgerA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-29T08:00:03.212Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "AppendRange_1", - "ActivityDisplayName": "Append Range: today's transactions", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ff333333-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681099001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/manifest.json deleted file mode 100644 index b52f9a2560..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Append Range ledger transaction append scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "c53ac0c2b8.json"}, - {"match": "or folders list --output json", "file": "c53ac0c2b8.json"}, - {"match": "or jobs list --folder-key \"f00aaaaa-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "84b50af139.json"}, - {"match": "or jobs list --folder-key f00aaaaa-3333-4444-5555-666677778888 --state Faulted --output json", "file": "84b50af139.json"}, - {"match": "or jobs get \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "ff43493cfb.json"}, - {"match": "or jobs get ff333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "ff43493cfb.json"}, - {"match": "or jobs history \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "d4b97a9106.json"}, - {"match": "or jobs history ff333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "d4b97a9106.json"}, - {"match": "or jobs logs \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "296981e65e.json"}, - {"match": "or jobs logs ff333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "296981e65e.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/uip-fixture.json new file mode 100644 index 0000000000..533a9f3294 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f00aaaaa-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-29T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-29T08:00:03.212Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelLedgerProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailyLedgerA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00aaaaa-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089).\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ff333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-29T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-29T08:00:03.212Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ff33333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089).\\r\\n at UiPath.Excel.Activities.Business.AppendRangeX.ValidateTargetRegion(Worksheet sheet, Int32 startRow, Int32 endRow)\\r\\n at UiPath.Excel.Activities.Business.AppendRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-29T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelLedgerProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681001,\r\n \"OrganizationUnitId\": 6100001,\r\n \"FolderKey\": \"f00aaaaa-3333-4444-5555-666677778888\",\r\n \"Reference\": \"DailyLedgerA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-29T08:00:03.212Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"AppendRange_1\",\r\n \"ActivityDisplayName\": \"Append Range: today's transactions\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681099001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ff333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-29T08:00:01.005Z\"},\r\n {\"Key\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-29T08:00:01.300Z\"},\r\n {\"Key\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-29T08:00:03.212Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ff333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ff3333cc-001\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-29T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-002\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-29T08:00:01.418Z\",\r\n \"Message\": \"[ExcelLedgerProcess] Appending today's transactions to C:\\\\Robot\\\\Data\\\\ledger.xlsx\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-003\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-29T08:00:01.910Z\",\r\n \"Message\": \"Invoke: FetchTransactions.xaml — returned dtTransactions with 78 rows, 6 columns (TxnId, TxnDate, Account, Amount, Currency, Status)\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-004\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-29T08:00:02.110Z\",\r\n \"Message\": \"Use Excel File: ledger.xlsx — workbook opened (OpenXML provider); UiPath.Excel.Activities version 2.24.7 (>= 2.8.5)\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-005\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-29T08:00:02.310Z\",\r\n \"Message\": \"Append Range: today's transactions — target sheet 'Transactions' inspection: AutoFilter active on column 'Status' (hiding rows where Status='Reconciled')\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-006\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-29T08:00:02.510Z\",\r\n \"Message\": \"Append Range: today's transactions — last data row on sheet 'Transactions': row 4100. Computed append region: rows 4101-4178 (78 rows for the dtTransactions payload).\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-007\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-29T08:00:02.710Z\",\r\n \"Message\": \"Append Range: today's transactions — target region validation (v2.8.5+): hidden rows detected in append target — sheet rows 4012-4089 are hidden by the AutoFilter and overlap with the computed append region's lookback. Pre-2.8.5 versions would have silently appended over the hidden region; v2.8.5+ throws to prevent data loss.\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff3333cc-008\",\r\n \"JobKey\": \"ff333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-29T08:00:03.202Z\",\r\n \"Message\": \"Append Range: today's transactions — UiPath.Excel.BusinessException: Cannot append to a range that contains hidden rows or columns. The target sheet 'Transactions' has hidden rows in the computed append region (rows 4012-4089).\",\r\n \"ProcessName\": \"ExcelLedgerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f00aaaaa-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"LedgerOps\",\r\n \"FullyQualifiedName\": \"LedgerOps\",\r\n \"Description\": \"Folder for daily transaction-ledger append automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6100001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/task.yaml index b3485c2199..8364dca48f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-hidden-rows-target/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/README.md new file mode 100644 index 0000000000..6f1b9c3586 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/README.md @@ -0,0 +1,57 @@ +# Excel Append Range — Activity Variant Mismatch (Missing Scope) + +This scenario reproduces an Append Range failure where the Modern +`AppendRangeX` activity is placed at the workflow's root without an +enclosing `Use Excel File` container. The job ends with: + +``` +UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's Append Range activity is the Modern +`AppendRangeX` surface, which requires a `Use Excel File` scope to +provide workbook context. In `Main.xaml` the activity sits at the +workflow's root — no surrounding scope. Scope validation detects the +missing container and throws before any provider call. + +The fix branches on the deployment context: if the Robot host has +Excel installed, wrap in `Use Excel File`; if not, switch surface to +the standalone `Append Range Workbook` activity. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/append-range-failures.md` +(the "Activity variant mismatch" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelPayrollProcess` project — `Invoke BuildPayrollRows.xaml` → `AppendRangeX` activity at the root with no enclosing scope | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException naming +the missing scope) → `jobs logs` (scope-validation Trace + the +BusinessException) → workflow source review (confirms no `Use Excel +File` wraps the AppendRange activity) → conclude branch 1. + +> **Note on the deployment-context fork.** The fix has two viable +> primary paths depending on whether Excel is installed on the +> Robot host. The test grades agents that recognize this fork +> and recommend the appropriate fix (or, more rigorously, ask the +> user about the host's Excel availability before picking one). +> Agents that recommend "just switch to Workbook" without +> considering host context get partial credit — Workbook is a +> legitimate alternative, but switching without understanding +> the surface differences is the playbook's anti-pattern. + +> **Note on fixtures.** Synthetic. The HR sub-workflow, DataTable +> shape, and exact BusinessException wording are placeholders. The +> test grades whether the agent identifies the missing scope as the +> root cause and recommends a structural fix (wrap in scope, or +> switch surface with host-context awareness). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/8e3e382b94.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/8e3e382b94.json deleted file mode 100644 index c17e9c5512..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/8e3e382b94.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-27T08:00:01.005Z"}, - {"Key": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-27T08:00:01.300Z"}, - {"Key": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-27T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/94432654a2.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/94432654a2.json deleted file mode 100644 index aa7d6984b1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/94432654a2.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-27T08:00:01.300Z", - "EndTime": "2026-05-27T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelPayrollProcess", - "Source": "Schedule", - "Reference": "WeeklyPayrollA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0088888-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/986e159c31.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/986e159c31.json deleted file mode 100644 index 89bd199ab5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/986e159c31.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-27T08:00:01.300Z", - "EndTime": "2026-05-27T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ff11111-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity.\r\n at UiPath.Excel.Activities.Business.AppendRangeX.ValidateScope(NativeActivityContext context)\r\n at UiPath.Excel.Activities.Business.AppendRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-27T08:00:01.005Z", - "ReleaseName": "ExcelPayrollProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680801, - "OrganizationUnitId": 6080001, - "FolderKey": "f0088888-3333-4444-5555-666677778888", - "Reference": "WeeklyPayrollA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "AppendRange_1", - "ActivityDisplayName": "Append Range: payroll rows", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680899001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/README.md deleted file mode 100644 index cbce4faf3d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Excel Append Range — Activity Variant Mismatch (Missing Scope) - -This scenario reproduces an Append Range failure where the Modern -`AppendRangeX` activity is placed at the workflow's root without an -enclosing `Use Excel File` container. The job ends with: - -``` -UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's Append Range activity is the Modern -`AppendRangeX` surface, which requires a `Use Excel File` scope to -provide workbook context. In `Main.xaml` the activity sits at the -workflow's root — no surrounding scope. Scope validation detects the -missing container and throws before any provider call. - -The fix branches on the deployment context: if the Robot host has -Excel installed, wrap in `Use Excel File`; if not, switch surface to -the standalone `Append Range Workbook` activity. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/append-range-failures.md` -(the "Activity variant mismatch" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelPayrollProcess` project — `Invoke BuildPayrollRows.xaml` → `AppendRangeX` activity at the root with no enclosing scope | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture the scope-validation log line that explicitly names the missing container, and the BusinessException wording names "Use Excel File" as the required scope | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException naming -the missing scope) → `jobs logs` (scope-validation Trace + the -BusinessException) → workflow source review (confirms no `Use Excel -File` wraps the AppendRange activity) → conclude branch 1. - -> **Note on the deployment-context fork.** The fix has two viable -> primary paths depending on whether Excel is installed on the -> Robot host. The test grades agents that recognize this fork -> and recommend the appropriate fix (or, more rigorously, ask the -> user about the host's Excel availability before picking one). -> Agents that recommend "just switch to Workbook" without -> considering host context get partial credit — Workbook is a -> legitimate alternative, but switching without understanding -> the surface differences is the playbook's anti-pattern. - -> **Note on fixtures.** Synthetic. The HR sub-workflow, DataTable -> shape, and exact BusinessException wording are placeholders. The -> test grades whether the agent identifies the missing scope as the -> root cause and recommends a structural fix (wrap in scope, or -> switch surface with host-context awareness). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/cc5437bd31.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/cc5437bd31.json deleted file mode 100644 index df57e1eda6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/cc5437bd31.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0088888-3333-4444-5555-666677778888", - "DisplayName": "PayrollOps", - "FullyQualifiedName": "PayrollOps", - "Description": "Folder for weekly payroll-ledger automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6080001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/fcfc48a7ad.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/fcfc48a7ad.json deleted file mode 100644 index 1d73545e3d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/fcfc48a7ad.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ff1111cc-001", - "JobKey": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-27T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelPayrollProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff1111cc-002", - "JobKey": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-27T08:00:01.418Z", - "Message": "[ExcelPayrollProcess] Appending current pay period rows to C:\\Robot\\Data\\payroll-ledger.xlsx", - "ProcessName": "ExcelPayrollProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff1111cc-003", - "JobKey": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-27T08:00:02.010Z", - "Message": "Invoke: BuildPayrollRows.xaml — returned dtPayroll with 47 rows, 7 columns (EmployeeId, FullName, Hours, RateUSD, Gross, TaxUSD, NetUSD)", - "ProcessName": "ExcelPayrollProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff1111cc-004", - "JobKey": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-27T08:00:02.310Z", - "Message": "Append Range: payroll rows — resolved config WorkbookPath='C:\\Robot\\Data\\payroll-ledger.xlsx' Sheet='Payroll' DataTable='dtPayroll' AddHeaders=False", - "ProcessName": "ExcelPayrollProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff1111cc-005", - "JobKey": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-27T08:00:02.510Z", - "Message": "Append Range: payroll rows — scope validation: no enclosing Use Excel File / Excel Application Scope found; this is the Modern AppendRangeX surface which requires a Use Excel File container", - "ProcessName": "ExcelPayrollProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ff1111cc-006", - "JobKey": "ff111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-27T08:00:02.802Z", - "Message": "Append Range: payroll rows — UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity.", - "ProcessName": "ExcelPayrollProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/manifest.json deleted file mode 100644 index f407125eec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Append Range payroll row append scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "cc5437bd31.json"}, - {"match": "or folders list --output json", "file": "cc5437bd31.json"}, - {"match": "or jobs list --folder-key \"f0088888-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "94432654a2.json"}, - {"match": "or jobs list --folder-key f0088888-3333-4444-5555-666677778888 --state Faulted --output json", "file": "94432654a2.json"}, - {"match": "or jobs get \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "986e159c31.json"}, - {"match": "or jobs get ff111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "986e159c31.json"}, - {"match": "or jobs history \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "8e3e382b94.json"}, - {"match": "or jobs history ff111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "8e3e382b94.json"}, - {"match": "or jobs logs \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "fcfc48a7ad.json"}, - {"match": "or jobs logs ff111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "fcfc48a7ad.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..27707e5cb8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0088888-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-27T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-27T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelPayrollProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"WeeklyPayrollA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0088888-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ff111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-27T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-27T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ff11111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity.\\r\\n at UiPath.Excel.Activities.Business.AppendRangeX.ValidateScope(NativeActivityContext context)\\r\\n at UiPath.Excel.Activities.Business.AppendRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-27T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelPayrollProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680801,\r\n \"OrganizationUnitId\": 6080001,\r\n \"FolderKey\": \"f0088888-3333-4444-5555-666677778888\",\r\n \"Reference\": \"WeeklyPayrollA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"AppendRange_1\",\r\n \"ActivityDisplayName\": \"Append Range: payroll rows\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680899001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ff111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-27T08:00:01.005Z\"},\r\n {\"Key\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-27T08:00:01.300Z\"},\r\n {\"Key\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-27T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ff111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ff1111cc-001\",\r\n \"JobKey\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-27T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelPayrollProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff1111cc-002\",\r\n \"JobKey\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-27T08:00:01.418Z\",\r\n \"Message\": \"[ExcelPayrollProcess] Appending current pay period rows to C:\\\\Robot\\\\Data\\\\payroll-ledger.xlsx\",\r\n \"ProcessName\": \"ExcelPayrollProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff1111cc-003\",\r\n \"JobKey\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-27T08:00:02.010Z\",\r\n \"Message\": \"Invoke: BuildPayrollRows.xaml — returned dtPayroll with 47 rows, 7 columns (EmployeeId, FullName, Hours, RateUSD, Gross, TaxUSD, NetUSD)\",\r\n \"ProcessName\": \"ExcelPayrollProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff1111cc-004\",\r\n \"JobKey\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-27T08:00:02.310Z\",\r\n \"Message\": \"Append Range: payroll rows — resolved config WorkbookPath='C:\\\\Robot\\\\Data\\\\payroll-ledger.xlsx' Sheet='Payroll' DataTable='dtPayroll' AddHeaders=False\",\r\n \"ProcessName\": \"ExcelPayrollProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff1111cc-005\",\r\n \"JobKey\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-27T08:00:02.510Z\",\r\n \"Message\": \"Append Range: payroll rows — scope validation: no enclosing Use Excel File / Excel Application Scope found; this is the Modern AppendRangeX surface which requires a Use Excel File container\",\r\n \"ProcessName\": \"ExcelPayrollProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ff1111cc-006\",\r\n \"JobKey\": \"ff111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-27T08:00:02.802Z\",\r\n \"Message\": \"Append Range: payroll rows — UiPath.Excel.BusinessException: The 'Append Range' activity must be placed inside a 'Use Excel File' container, which manages the workbook context for the activity.\",\r\n \"ProcessName\": \"ExcelPayrollProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0088888-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"PayrollOps\",\r\n \"FullyQualifiedName\": \"PayrollOps\",\r\n \"Description\": \"Folder for weekly payroll-ledger automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6080001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/task.yaml index 15a34f516a..5986eb51e1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-append-range-variant-mismatch/task.yaml @@ -26,21 +26,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/README.md new file mode 100644 index 0000000000..782a18c150 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/README.md @@ -0,0 +1,63 @@ +# Excel Application Scope — Excel Not Installed on Host + +This scenario reproduces a Classic `Excel Application Scope` failure +where the Robot host has no Microsoft Excel installed. Classic scope +is COM-only; without `Excel.Application` registered in the host's +COM registry, the scope's acquisition step throws. The job ends with: + +``` +UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. +``` + +(with inner `COMException 0x80040154 REGDB_E_CLASSNOTREG`) + +## What this scenario uncovers + +**Root Cause:** The workflow uses Classic `Excel Application Scope` +which always requires Excel COM. The unattended Robot host has no +Excel installed (no registry entry under `HKLM:\Software\Microsoft\ +Office\*\Excel\InstallRoot`, no `excel.exe` on the filesystem). The +scope's `CoCreateInstance` for `CLSID {00024500-...}` (Excel. +Application) returns `REGDB_E_CLASSNOTREG`, surfaced as the +canonical "Make sure Excel is installed" BusinessException. + +The fix branches based on whether the workflow actually needs Excel +COM features: workflows that just read/write data can migrate to +Workbook activities or Modern `Use Excel File` (OpenXML default); +workflows that need COM features need Excel installed on the host. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-not-installed.md` +(the Classic `Excel Application Scope` discriminator). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelInvoiceReport` project — Classic `` wrapping a `Read Range`. The workflow doesn't actually need COM features (just reads data) — the Classic surface choice is the problem | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException with +inner COMException 0x80040154) → `jobs logs` (the three Trace lines +pinning surface + host state + COM error) → workflow source review +(confirms Classic surface used for a workflow that doesn't need +COM) → conclude the missing-install cause through the Classic-scope discriminator. + +> **Note on the surface-choice fork.** The test rewards agents +> who recognize that the fix depends on whether the workflow +> needs Excel features. An agent that recommends ONLY "install +> Excel on the host" without considering the alternative +> (migrate to Workbook surface) gets partial credit. An agent +> that recognizes the fork and asks (or recommends both paths +> with their applicable conditions) scores full marks. + +> **Note on fixtures.** Synthetic. The exact COM HRESULT, +> registry path, and host detection logic are placeholders. The +> test grades whether the agent identifies the surface-vs-host +> mismatch AND recommends a viable fix (install Excel, +> repair Excel, migrate to Workbook, or migrate to Modern +> OpenXML). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/2d646a7df9.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/2d646a7df9.json deleted file mode 100644 index 115b6a7564..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/2d646a7df9.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T08:00:01.300Z", - "EndTime": "2026-05-30T08:00:02.512Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.\r\n at UiPath.Excel.Activities.ExcelApplicationScope.AcquireExcelComServer()\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\r\nInner: System.Runtime.InteropServices.COMException (0x80040154): Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (0x80040154 REGDB_E_CLASSNOTREG).", - "CreationTime": "2026-05-30T08:00:01.005Z", - "ReleaseName": "ExcelInvoiceReport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681101, - "OrganizationUnitId": 6110001, - "FolderKey": "f00bbbbb-3333-4444-5555-666677778888", - "Reference": "WeeklyInvoiceA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T08:00:02.512Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelApplicationScope_1", - "ActivityDisplayName": "Excel Application Scope: invoice-report.xlsm", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681199001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/47d6a74a1d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/47d6a74a1d.json deleted file mode 100644 index 470e807984..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/47d6a74a1d.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa1111cc-001", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelInvoiceReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-002", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T08:00:01.418Z", - "Message": "[ExcelInvoiceReport] Opening C:\\Robot\\Data\\invoice-report.xlsm to read this week's invoice data", - "ProcessName": "ExcelInvoiceReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-003", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T08:00:01.610Z", - "Message": "Excel Application Scope: invoice-report.xlsm — surface: Classic ExcelApplicationScope (COM-only). Attempting to acquire Excel.Application COM server.", - "ProcessName": "ExcelInvoiceReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-004", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T08:00:01.810Z", - "Message": "Excel Application Scope: invoice-report.xlsm — CLSID {00024500-0000-0000-C000-000000000046} (Excel.Application) not registered on host. Host: MOCK-HOST. Robot user: UIPATH\\AUTOMATION1. Excel install state: NOT INSTALLED (no entry under HKLM:\\Software\\Microsoft\\Office\\*\\Excel\\InstallRoot, no excel.exe on PATH or under Program Files).", - "ProcessName": "ExcelInvoiceReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-005", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-30T08:00:02.502Z", - "Message": "Excel Application Scope: invoice-report.xlsm — UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. Inner: COMException 0x80040154 REGDB_E_CLASSNOTREG (Class not registered).", - "ProcessName": "ExcelInvoiceReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/61f4570ec9.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/61f4570ec9.json deleted file mode 100644 index 3ee3a018c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/61f4570ec9.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-30T08:00:01.005Z"}, - {"Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-30T08:00:01.300Z"}, - {"Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-30T08:00:02.512Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/89d2086078.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/89d2086078.json deleted file mode 100644 index df128a8e31..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/89d2086078.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f00bbbbb-3333-4444-5555-666677778888", - "DisplayName": "InvoiceReports", - "FullyQualifiedName": "InvoiceReports", - "Description": "Folder for weekly invoice-report automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6110001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/9243450e08.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/9243450e08.json deleted file mode 100644 index d1b9dfc42a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/9243450e08.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T08:00:01.300Z", - "EndTime": "2026-05-30T08:00:02.512Z", - "State": "Faulted", - "ReleaseName": "ExcelInvoiceReport", - "Source": "Schedule", - "Reference": "WeeklyInvoiceA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00bbbbb-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/README.md deleted file mode 100644 index 03a2386070..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# Excel Application Scope — Excel Not Installed on Host - -This scenario reproduces a Classic `Excel Application Scope` failure -where the Robot host has no Microsoft Excel installed. Classic scope -is COM-only; without `Excel.Application` registered in the host's -COM registry, the scope's acquisition step throws. The job ends with: - -``` -UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. -``` - -(with inner `COMException 0x80040154 REGDB_E_CLASSNOTREG`) - -## What this scenario uncovers - -**Root Cause:** The workflow uses Classic `Excel Application Scope` -which always requires Excel COM. The unattended Robot host has no -Excel installed (no registry entry under `HKLM:\Software\Microsoft\ -Office\*\Excel\InstallRoot`, no `excel.exe` on the filesystem). The -scope's `CoCreateInstance` for `CLSID {00024500-...}` (Excel. -Application) returns `REGDB_E_CLASSNOTREG`, surfaced as the -canonical "Make sure Excel is installed" BusinessException. - -The fix branches based on whether the workflow actually needs Excel -COM features: workflows that just read/write data can migrate to -Workbook activities or Modern `Use Excel File` (OpenXML default); -workflows that need COM features need Excel installed on the host. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-not-installed.md` -(the Classic `Excel Application Scope` discriminator). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelInvoiceReport` project — Classic `` wrapping a `Read Range`. The workflow doesn't actually need COM features (just reads data) — the Classic surface choice is the problem | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture (a) the surface choice (Classic, COM-only), (b) the host's install state (no registry entry, no `excel.exe`), and (c) the canonical COM `REGDB_E_CLASSNOTREG` HRESULT | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException with -inner COMException 0x80040154) → `jobs logs` (the three Trace lines -pinning surface + host state + COM error) → workflow source review -(confirms Classic surface used for a workflow that doesn't need -COM) → conclude the missing-install cause through the Classic-scope discriminator. - -> **Note on the surface-choice fork.** The test rewards agents -> who recognize that the fix depends on whether the workflow -> needs Excel features. An agent that recommends ONLY "install -> Excel on the host" without considering the alternative -> (migrate to Workbook surface) gets partial credit. An agent -> that recognizes the fork and asks (or recommends both paths -> with their applicable conditions) scores full marks. - -> **Note on fixtures.** Synthetic. The exact COM HRESULT, -> registry path, and host detection logic are placeholders. The -> test grades whether the agent identifies the surface-vs-host -> mismatch AND recommends a viable fix (install Excel, -> repair Excel, migrate to Workbook, or migrate to Modern -> OpenXML). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/manifest.json deleted file mode 100644 index 69d4c910a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Application Scope invoice-report scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "89d2086078.json"}, - {"match": "or folders list --output json", "file": "89d2086078.json"}, - {"match": "or jobs list --folder-key \"f00bbbbb-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "9243450e08.json"}, - {"match": "or jobs list --folder-key f00bbbbb-3333-4444-5555-666677778888 --state Faulted --output json", "file": "9243450e08.json"}, - {"match": "or jobs get \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "2d646a7df9.json"}, - {"match": "or jobs get aa111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "2d646a7df9.json"}, - {"match": "or jobs history \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "61f4570ec9.json"}, - {"match": "or jobs history aa111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "61f4570ec9.json"}, - {"match": "or jobs logs \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "47d6a74a1d.json"}, - {"match": "or jobs logs aa111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "47d6a74a1d.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/uip-fixture.json new file mode 100644 index 0000000000..2562fef8c3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f00bbbbb-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-30T08:00:02.512Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelInvoiceReport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"WeeklyInvoiceA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00bbbbb-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-30T08:00:02.512Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.AcquireExcelComServer()\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\\r\\nInner: System.Runtime.InteropServices.COMException (0x80040154): Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (0x80040154 REGDB_E_CLASSNOTREG).\",\r\n \"CreationTime\": \"2026-05-30T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelInvoiceReport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681101,\r\n \"OrganizationUnitId\": 6110001,\r\n \"FolderKey\": \"f00bbbbb-3333-4444-5555-666677778888\",\r\n \"Reference\": \"WeeklyInvoiceA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T08:00:02.512Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelApplicationScope_1\",\r\n \"ActivityDisplayName\": \"Excel Application Scope: invoice-report.xlsm\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681199001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-30T08:00:01.005Z\"},\r\n {\"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-30T08:00:01.300Z\"},\r\n {\"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-30T08:00:02.512Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa1111cc-001\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelInvoiceReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-002\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T08:00:01.418Z\",\r\n \"Message\": \"[ExcelInvoiceReport] Opening C:\\\\Robot\\\\Data\\\\invoice-report.xlsm to read this week's invoice data\",\r\n \"ProcessName\": \"ExcelInvoiceReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-003\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T08:00:01.610Z\",\r\n \"Message\": \"Excel Application Scope: invoice-report.xlsm — surface: Classic ExcelApplicationScope (COM-only). Attempting to acquire Excel.Application COM server.\",\r\n \"ProcessName\": \"ExcelInvoiceReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-004\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T08:00:01.810Z\",\r\n \"Message\": \"Excel Application Scope: invoice-report.xlsm — CLSID {00024500-0000-0000-C000-000000000046} (Excel.Application) not registered on host. Host: MOCK-HOST. Robot user: UIPATH\\\\AUTOMATION1. Excel install state: NOT INSTALLED (no entry under HKLM:\\\\Software\\\\Microsoft\\\\Office\\\\*\\\\Excel\\\\InstallRoot, no excel.exe on PATH or under Program Files).\",\r\n \"ProcessName\": \"ExcelInvoiceReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-005\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-30T08:00:02.502Z\",\r\n \"Message\": \"Excel Application Scope: invoice-report.xlsm — UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. Inner: COMException 0x80040154 REGDB_E_CLASSNOTREG (Class not registered).\",\r\n \"ProcessName\": \"ExcelInvoiceReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f00bbbbb-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"InvoiceReports\",\r\n \"FullyQualifiedName\": \"InvoiceReports\",\r\n \"Description\": \"Folder for weekly invoice-report automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6110001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/task.yaml index 9b9a674d41..fcbacd2ccf 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-excel-not-installed/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/README.md new file mode 100644 index 0000000000..3f8447fa37 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/README.md @@ -0,0 +1,57 @@ +# Use Excel File — COM / RPC Race Across Sequential Scopes + +This scenario reproduces a Modern `Use Excel File` failure where two +sequential cards (with `ReadFormatting=True` COM-forcing) race on +the EXCEL.EXE process lifecycle. The first card's scope-end +terminates EXCEL.EXE; the second card tries to reuse the COM +apartment from the dead process. The job ends with: + +``` +System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED) +``` + +## What this scenario uncovers + +**Root Cause:** Two `Use Excel File` cards in sequence, both with +`ReadFormatting=True` forcing COM fallback, with NO outer `Excel +Process Scope` wrapping them. Card 1's scope-end policy terminates +EXCEL.EXE (PID 8472) because there's no Process Scope governing +the lifecycle. Card 2's COM acquisition tries to reuse the dead +process's apartment and hits RPC_E_DISCONNECTED. + +The canonical fix is wrapping both cards in an `Excel Process +Scope` — the outer container that governs EXCEL.EXE lifecycle +across inner cards. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-card-failures.md` +(the "COM / RPC failures across scopes" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelMultiReportProcess` project — two sequential `` cards (sales-q1.xlsm + sales-q2.xlsm), both with `ReadFormatting=True`, NO outer `` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (COMException 0x80010108 +RPC_E_DISCONNECTED on UseExcelFile_2) → `jobs logs` (PID 8472 +lifecycle chain naming the missing Excel Process Scope) → workflow +source review (confirms no outer Process Scope) → conclude branch 3. + +> **Note on the anti-pattern trap.** The natural temptation is to +> add a `Delay` between the two cards. The playbook explicitly +> rejects this — the Delay sometimes works by coincidence but the +> race is still there. The test penalizes Delay-as-fix +> recommendations and rewards the Process Scope wrap. + +> **Note on fixtures.** Synthetic. The exact PID, COM apartment +> details, and scope-policy log wording are placeholders. The +> test grades whether the agent identifies the missing Process +> Scope as the root cause AND recommends the Process Scope wrap +> (or one of the alternatives: remove COM-forcing properties, +> consolidate workbooks). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/2aaed5f64d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/2aaed5f64d.json deleted file mode 100644 index 35c7ae894d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/2aaed5f64d.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T09:00:01.300Z", - "EndTime": "2026-05-30T09:00:08.412Z", - "State": "Faulted", - "ReleaseName": "ExcelMultiReportProcess", - "Source": "Schedule", - "Reference": "QuarterlyComparisonA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00ccccc-3333-4444-5555-666677778888", - "Info": "System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED)", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/3dec41f0e2.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/3dec41f0e2.json deleted file mode 100644 index aab79f2bfe..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/3dec41f0e2.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f00ccccc-3333-4444-5555-666677778888", - "DisplayName": "QuarterlyReports", - "FullyQualifiedName": "QuarterlyReports", - "Description": "Folder for quarterly comparison-report automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6120001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/7a13867de5.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/7a13867de5.json deleted file mode 100644 index bcd718806e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/7a13867de5.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa2222cc-001", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T09:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-002", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T09:00:01.418Z", - "Message": "[ExcelMultiReportProcess] Reading Q1 and Q2 sales workbooks for the quarterly comparison report", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-003", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:00:01.610Z", - "Message": "Use Excel File: sales-q1.xlsm — ReadFormatting=True forces COM fallback (OpenXML provider declined). Launching new EXCEL.EXE process (PID 8472).", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-004", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:00:03.110Z", - "Message": "Read Range: Q1 sales — returned 1247 rows. Body of UseExcelFile_1 complete.", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-005", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:00:03.510Z", - "Message": "Use Excel File: sales-q1.xlsm — scope exiting. No outer Excel Process Scope detected: terminating EXCEL.EXE PID 8472 per scope-owned-lifecycle policy.", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-006", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:00:04.210Z", - "Message": "Use Excel File: sales-q2.xlsm — ReadFormatting=True forces COM fallback. Attempting to reuse Excel COM apartment from previous scope (PID 8472).", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-007", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:00:07.910Z", - "Message": "Use Excel File: sales-q2.xlsm — COM apartment for PID 8472 is unreachable (process terminated). HRESULT 0x80010108 RPC_E_DISCONNECTED.", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa2222cc-008", - "JobKey": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-30T09:00:08.402Z", - "Message": "Use Excel File: sales-q2.xlsm — System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED)", - "ProcessName": "ExcelMultiReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/7c3905943b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/7c3905943b.json deleted file mode 100644 index 4e5cb13ffd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/7c3905943b.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-30T09:00:01.005Z"}, - {"Key": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-30T09:00:01.300Z"}, - {"Key": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-30T09:00:08.412Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/README.md deleted file mode 100644 index 5b1caeee90..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Use Excel File — COM / RPC Race Across Sequential Scopes - -This scenario reproduces a Modern `Use Excel File` failure where two -sequential cards (with `ReadFormatting=True` COM-forcing) race on -the EXCEL.EXE process lifecycle. The first card's scope-end -terminates EXCEL.EXE; the second card tries to reuse the COM -apartment from the dead process. The job ends with: - -``` -System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED) -``` - -## What this scenario uncovers - -**Root Cause:** Two `Use Excel File` cards in sequence, both with -`ReadFormatting=True` forcing COM fallback, with NO outer `Excel -Process Scope` wrapping them. Card 1's scope-end policy terminates -EXCEL.EXE (PID 8472) because there's no Process Scope governing -the lifecycle. Card 2's COM acquisition tries to reuse the dead -process's apartment and hits RPC_E_DISCONNECTED. - -The canonical fix is wrapping both cards in an `Excel Process -Scope` — the outer container that governs EXCEL.EXE lifecycle -across inner cards. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-card-failures.md` -(the "COM / RPC failures across scopes" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelMultiReportProcess` project — two sequential `` cards (sales-q1.xlsm + sales-q2.xlsm), both with `ReadFormatting=True`, NO outer `` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries trace the EXCEL.EXE PID 8472 lifecycle: launch → first scope completes → "No outer Excel Process Scope detected: terminating EXCEL.EXE PID 8472" → second scope tries to reuse the gone apartment → RPC_E_DISCONNECTED | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (COMException 0x80010108 -RPC_E_DISCONNECTED on UseExcelFile_2) → `jobs logs` (PID 8472 -lifecycle chain naming the missing Excel Process Scope) → workflow -source review (confirms no outer Process Scope) → conclude branch 3. - -> **Note on the anti-pattern trap.** The natural temptation is to -> add a `Delay` between the two cards. The playbook explicitly -> rejects this — the Delay sometimes works by coincidence but the -> race is still there. The test penalizes Delay-as-fix -> recommendations and rewards the Process Scope wrap. - -> **Note on fixtures.** Synthetic. The exact PID, COM apartment -> details, and scope-policy log wording are placeholders. The -> test grades whether the agent identifies the missing Process -> Scope as the root cause AND recommends the Process Scope wrap -> (or one of the alternatives: remove COM-forcing properties, -> consolidate workbooks). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/e562530d9a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/e562530d9a.json deleted file mode 100644 index d46643f70f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/e562530d9a.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T09:00:01.300Z", - "EndTime": "2026-05-30T09:00:08.412Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa22222-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED)\r\n at UiPath.Excel.Activities.Business.UseExcelFile.AcquireWorkbook(String path)\r\n at UiPath.Excel.Activities.Business.UseExcelFile.OnExecute(NativeActivityContext context)\r\nInner: COM apartment for prior Excel.Application process (PID 8472) is no longer reachable; the process terminated when UseExcelFile_1 body completed.", - "CreationTime": "2026-05-30T09:00:01.005Z", - "ReleaseName": "ExcelMultiReportProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681201, - "OrganizationUnitId": 6120001, - "FolderKey": "f00ccccc-3333-4444-5555-666677778888", - "Reference": "QuarterlyComparisonA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T09:00:08.412Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "ActivityName": "UseExcelFile_2", - "ActivityDisplayName": "Use Excel File: sales-q2.xlsm", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa222222-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681299001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/manifest.json deleted file mode 100644 index 9c8fd58f57..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Use Excel File multi-quarter report scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "3dec41f0e2.json"}, - {"match": "or folders list --output json", "file": "3dec41f0e2.json"}, - {"match": "or jobs list --folder-key \"f00ccccc-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "2aaed5f64d.json"}, - {"match": "or jobs list --folder-key f00ccccc-3333-4444-5555-666677778888 --state Faulted --output json", "file": "2aaed5f64d.json"}, - {"match": "or jobs get \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "e562530d9a.json"}, - {"match": "or jobs get aa222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "e562530d9a.json"}, - {"match": "or jobs history \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "7c3905943b.json"}, - {"match": "or jobs history aa222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "7c3905943b.json"}, - {"match": "or jobs logs \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "7a13867de5.json"}, - {"match": "or jobs logs aa222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "7a13867de5.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/uip-fixture.json new file mode 100644 index 0000000000..23610bcc94 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f00ccccc-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T09:00:01.300Z\",\r\n \"EndTime\": \"2026-05-30T09:00:08.412Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelMultiReportProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"QuarterlyComparisonA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00ccccc-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED)\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T09:00:01.300Z\",\r\n \"EndTime\": \"2026-05-30T09:00:08.412Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa22222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED)\\r\\n at UiPath.Excel.Activities.Business.UseExcelFile.AcquireWorkbook(String path)\\r\\n at UiPath.Excel.Activities.Business.UseExcelFile.OnExecute(NativeActivityContext context)\\r\\nInner: COM apartment for prior Excel.Application process (PID 8472) is no longer reachable; the process terminated when UseExcelFile_1 body completed.\",\r\n \"CreationTime\": \"2026-05-30T09:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelMultiReportProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681201,\r\n \"OrganizationUnitId\": 6120001,\r\n \"FolderKey\": \"f00ccccc-3333-4444-5555-666677778888\",\r\n \"Reference\": \"QuarterlyComparisonA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T09:00:08.412Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"ActivityName\": \"UseExcelFile_2\",\r\n \"ActivityDisplayName\": \"Use Excel File: sales-q2.xlsm\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681299001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-30T09:00:01.005Z\"},\r\n {\"Key\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-30T09:00:01.300Z\"},\r\n {\"Key\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-30T09:00:08.412Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa2222cc-001\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T09:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-002\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T09:00:01.418Z\",\r\n \"Message\": \"[ExcelMultiReportProcess] Reading Q1 and Q2 sales workbooks for the quarterly comparison report\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-003\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:00:01.610Z\",\r\n \"Message\": \"Use Excel File: sales-q1.xlsm — ReadFormatting=True forces COM fallback (OpenXML provider declined). Launching new EXCEL.EXE process (PID 8472).\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-004\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:00:03.110Z\",\r\n \"Message\": \"Read Range: Q1 sales — returned 1247 rows. Body of UseExcelFile_1 complete.\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-005\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:00:03.510Z\",\r\n \"Message\": \"Use Excel File: sales-q1.xlsm — scope exiting. No outer Excel Process Scope detected: terminating EXCEL.EXE PID 8472 per scope-owned-lifecycle policy.\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-006\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:00:04.210Z\",\r\n \"Message\": \"Use Excel File: sales-q2.xlsm — ReadFormatting=True forces COM fallback. Attempting to reuse Excel COM apartment from previous scope (PID 8472).\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-007\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:00:07.910Z\",\r\n \"Message\": \"Use Excel File: sales-q2.xlsm — COM apartment for PID 8472 is unreachable (process terminated). HRESULT 0x80010108 RPC_E_DISCONNECTED.\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa2222cc-008\",\r\n \"JobKey\": \"aa222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-30T09:00:08.402Z\",\r\n \"Message\": \"Use Excel File: sales-q2.xlsm — System.Runtime.InteropServices.COMException (0x80010108): The object invoked has disconnected from its clients. (RPC_E_DISCONNECTED)\",\r\n \"ProcessName\": \"ExcelMultiReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f00ccccc-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"QuarterlyReports\",\r\n \"FullyQualifiedName\": \"QuarterlyReports\",\r\n \"Description\": \"Folder for quarterly comparison-report automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6120001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/task.yaml index b37908053c..90a1b04281 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-rpc-race-across-scopes/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/README.md new file mode 100644 index 0000000000..5a994a5da6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/README.md @@ -0,0 +1,64 @@ +# Use Excel File — Microsoft Purview Sensitivity Label Rejection + +This scenario reproduces a Modern `Use Excel File` card failure +where the target workbook has a Microsoft Purview sensitivity label +applied (`Confidential - Client Data`) but the card's +`SensitivityLabel` / `SensitivityOperation` properties (available +in `UiPath.Excel.Activities` v2.23.4+) are not configured. The card +detects the label, validates against the unset properties, and +refuses the write. The job ends with: + +``` +UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured. +``` + +## What this scenario uncovers + +**Root Cause:** The workbook has a Microsoft Purview label applied +(`Confidential - Client Data`, label ID +`8c2f3a4d-1234-5678-abcd-a17000c11e001`). Package version 2.24.7 +supports the SensitivityLabel / SensitivityOperation card properties +(introduced in v2.23.4), but the workflow doesn't configure either. +Under v2.23.4+ semantics, write operations against labeled workbooks +require explicit acknowledgement (Keep / Apply / Remove). The card +throws BEFORE the inner Write Range runs. + +The canonical fix is `SensitivityLabel="Confidential - Client Data"` ++ `SensitivityOperation=Keep` on the card (preserves the existing +label while writing data). + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-card-failures.md` +(the "Sensitivity label rejection" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelClientReportProcess` project — `` with no `SensitivityLabel` / `SensitivityOperation` properties, wrapping a `` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException naming +the Purview label) → `jobs logs` (the four-line chain pinning +package version + label detection + unset card props + exception) +→ workflow source review (confirms no label properties on the +card) → conclude branch 5. + +> **Note on the anti-pattern trap.** The natural temptation is to +> downgrade `UiPath.Excel.Activities` to a pre-v2.23.4 version to +> silence the error. The playbook explicitly rejects this — +> pre-v2.23.4 packages handle labeled workbooks INCORRECTLY +> (silent label-stripping = compliance violation). The test +> penalizes the package-downgrade recommendation and rewards +> the configure-the-properties fix. + +> **Note on fixtures.** Synthetic. The label name, label ID, and +> Purview-detection log format are placeholders. The test grades +> whether the agent identifies the label-on-workbook + +> unset-properties combination as the root cause AND recommends +> the SensitivityLabel + SensitivityOperation configuration fix +> (NOT package downgrade, NOT bare Try-Catch). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/19067a612e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/19067a612e.json deleted file mode 100644 index 7c001e0cbc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/19067a612e.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-30T10:00:01.005Z"}, - {"Key": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-30T10:00:01.300Z"}, - {"Key": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-30T10:00:04.612Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/57ef23ff3c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/57ef23ff3c.json deleted file mode 100644 index 52f58f9fb9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/57ef23ff3c.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa3333cc-001", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T10:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa3333cc-002", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T10:00:01.418Z", - "Message": "[ExcelClientReportProcess] Writing client metrics to C:\\Robot\\Data\\client-report.xlsx", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa3333cc-003", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T10:00:01.910Z", - "Message": "Invoke: ComputeClientMetrics.xaml — returned dtClientMetrics with 89 rows, 6 columns (ClientId, ClientName, Revenue, ChurnRisk, NPSScore, RenewalDate)", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa3333cc-004", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T10:00:02.210Z", - "Message": "Use Excel File: client-report.xlsx — opened workbook (OpenXML provider). Package version: UiPath.Excel.Activities 2.24.7 (>= 2.23.4: SensitivityLabel and SensitivityOperation properties available).", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa3333cc-005", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T10:00:02.510Z", - "Message": "Use Excel File: client-report.xlsx — Microsoft Purview sensitivity label detected on workbook: 'Confidential - Client Data' (label ID: 8c2f3a4d-1234-5678-abcd-a17000c11e001). This label requires programmatic acknowledgement before write operations.", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa3333cc-006", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T10:00:02.810Z", - "Message": "Use Excel File: client-report.xlsx — card property inspection: SensitivityLabel=, SensitivityOperation=. Both properties are required for write operations against labeled workbooks under v2.23.4+.", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa3333cc-007", - "JobKey": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-30T10:00:04.602Z", - "Message": "Use Excel File: client-report.xlsx — UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured.", - "ProcessName": "ExcelClientReportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/README.md deleted file mode 100644 index 07127139e3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# Use Excel File — Microsoft Purview Sensitivity Label Rejection - -This scenario reproduces a Modern `Use Excel File` card failure -where the target workbook has a Microsoft Purview sensitivity label -applied (`Confidential - Client Data`) but the card's -`SensitivityLabel` / `SensitivityOperation` properties (available -in `UiPath.Excel.Activities` v2.23.4+) are not configured. The card -detects the label, validates against the unset properties, and -refuses the write. The job ends with: - -``` -UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured. -``` - -## What this scenario uncovers - -**Root Cause:** The workbook has a Microsoft Purview label applied -(`Confidential - Client Data`, label ID -`8c2f3a4d-1234-5678-abcd-a17000c11e001`). Package version 2.24.7 -supports the SensitivityLabel / SensitivityOperation card properties -(introduced in v2.23.4), but the workflow doesn't configure either. -Under v2.23.4+ semantics, write operations against labeled workbooks -require explicit acknowledgement (Keep / Apply / Remove). The card -throws BEFORE the inner Write Range runs. - -The canonical fix is `SensitivityLabel="Confidential - Client Data"` -+ `SensitivityOperation=Keep` on the card (preserves the existing -label while writing data). - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-card-failures.md` -(the "Sensitivity label rejection" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelClientReportProcess` project — `` with no `SensitivityLabel` / `SensitivityOperation` properties, wrapping a `` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture (a) the package version (≥2.23.4: properties available), (b) the workbook's Purview label detection naming the label ID, (c) the card-property inspection showing both label props ``, and (d) the BusinessException | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException naming -the Purview label) → `jobs logs` (the four-line chain pinning -package version + label detection + unset card props + exception) -→ workflow source review (confirms no label properties on the -card) → conclude branch 5. - -> **Note on the anti-pattern trap.** The natural temptation is to -> downgrade `UiPath.Excel.Activities` to a pre-v2.23.4 version to -> silence the error. The playbook explicitly rejects this — -> pre-v2.23.4 packages handle labeled workbooks INCORRECTLY -> (silent label-stripping = compliance violation). The test -> penalizes the package-downgrade recommendation and rewards -> the configure-the-properties fix. - -> **Note on fixtures.** Synthetic. The label name, label ID, and -> Purview-detection log format are placeholders. The test grades -> whether the agent identifies the label-on-workbook + -> unset-properties combination as the root cause AND recommends -> the SensitivityLabel + SensitivityOperation configuration fix -> (NOT package downgrade, NOT bare Try-Catch). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/b99242ce80.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/b99242ce80.json deleted file mode 100644 index 0a3335d27b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/b99242ce80.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f00ddddd-3333-4444-5555-666677778888", - "DisplayName": "ClientReports", - "FullyQualifiedName": "ClientReports", - "Description": "Folder for monthly client-report automations against AIP-protected workbooks.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6130001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/d819771869.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/d819771869.json deleted file mode 100644 index 6ae91ec02c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/d819771869.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T10:00:01.300Z", - "EndTime": "2026-05-30T10:00:04.612Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa33333-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured.\r\n at UiPath.Excel.Activities.Business.UseExcelFile.ValidateSensitivityLabel(WorkbookLabel detected, SensitivityLabelOperation configured)\r\n at UiPath.Excel.Activities.Business.UseExcelFile.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-30T10:00:01.005Z", - "ReleaseName": "ExcelClientReportProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681301, - "OrganizationUnitId": 6130001, - "FolderKey": "f00ddddd-3333-4444-5555-666677778888", - "Reference": "MonthlyClientReportA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T10:00:04.612Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "UseExcelFile_1", - "ActivityDisplayName": "Use Excel File: client-report.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681399001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/e0ae5a1555.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/e0ae5a1555.json deleted file mode 100644 index 6e6d224ea6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/e0ae5a1555.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T10:00:01.300Z", - "EndTime": "2026-05-30T10:00:04.612Z", - "State": "Faulted", - "ReleaseName": "ExcelClientReportProcess", - "Source": "Schedule", - "Reference": "MonthlyClientReportA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00ddddd-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/manifest.json deleted file mode 100644 index d2ab1fb3ab..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Use Excel File client-report scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "b99242ce80.json"}, - {"match": "or folders list --output json", "file": "b99242ce80.json"}, - {"match": "or jobs list --folder-key \"f00ddddd-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "e0ae5a1555.json"}, - {"match": "or jobs list --folder-key f00ddddd-3333-4444-5555-666677778888 --state Faulted --output json", "file": "e0ae5a1555.json"}, - {"match": "or jobs get \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "d819771869.json"}, - {"match": "or jobs get aa333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "d819771869.json"}, - {"match": "or jobs history \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "19067a612e.json"}, - {"match": "or jobs history aa333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "19067a612e.json"}, - {"match": "or jobs logs \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "57ef23ff3c.json"}, - {"match": "or jobs logs aa333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "57ef23ff3c.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/uip-fixture.json new file mode 100644 index 0000000000..1adc575cd8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f00ddddd-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T10:00:01.300Z\",\r\n \"EndTime\": \"2026-05-30T10:00:04.612Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelClientReportProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"MonthlyClientReportA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00ddddd-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T10:00:01.300Z\",\r\n \"EndTime\": \"2026-05-30T10:00:04.612Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa33333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured.\\r\\n at UiPath.Excel.Activities.Business.UseExcelFile.ValidateSensitivityLabel(WorkbookLabel detected, SensitivityLabelOperation configured)\\r\\n at UiPath.Excel.Activities.Business.UseExcelFile.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-30T10:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelClientReportProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681301,\r\n \"OrganizationUnitId\": 6130001,\r\n \"FolderKey\": \"f00ddddd-3333-4444-5555-666677778888\",\r\n \"Reference\": \"MonthlyClientReportA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T10:00:04.612Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"UseExcelFile_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: client-report.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681399001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-30T10:00:01.005Z\"},\r\n {\"Key\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-30T10:00:01.300Z\"},\r\n {\"Key\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-30T10:00:04.612Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa3333cc-001\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T10:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa3333cc-002\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T10:00:01.418Z\",\r\n \"Message\": \"[ExcelClientReportProcess] Writing client metrics to C:\\\\Robot\\\\Data\\\\client-report.xlsx\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa3333cc-003\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T10:00:01.910Z\",\r\n \"Message\": \"Invoke: ComputeClientMetrics.xaml — returned dtClientMetrics with 89 rows, 6 columns (ClientId, ClientName, Revenue, ChurnRisk, NPSScore, RenewalDate)\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa3333cc-004\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T10:00:02.210Z\",\r\n \"Message\": \"Use Excel File: client-report.xlsx — opened workbook (OpenXML provider). Package version: UiPath.Excel.Activities 2.24.7 (>= 2.23.4: SensitivityLabel and SensitivityOperation properties available).\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa3333cc-005\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T10:00:02.510Z\",\r\n \"Message\": \"Use Excel File: client-report.xlsx — Microsoft Purview sensitivity label detected on workbook: 'Confidential - Client Data' (label ID: 8c2f3a4d-1234-5678-abcd-a17000c11e001). This label requires programmatic acknowledgement before write operations.\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa3333cc-006\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T10:00:02.810Z\",\r\n \"Message\": \"Use Excel File: client-report.xlsx — card property inspection: SensitivityLabel=, SensitivityOperation=. Both properties are required for write operations against labeled workbooks under v2.23.4+.\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa3333cc-007\",\r\n \"JobKey\": \"aa333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-30T10:00:04.602Z\",\r\n \"Message\": \"Use Excel File: client-report.xlsx — UiPath.Excel.BusinessException: The workbook is protected by a Microsoft Purview sensitivity label ('Confidential - Client Data') that requires explicit handling. The SensitivityLabel and SensitivityOperation properties on the Use Excel File activity are not configured.\",\r\n \"ProcessName\": \"ExcelClientReportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f00ddddd-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"ClientReports\",\r\n \"FullyQualifiedName\": \"ClientReports\",\r\n \"Description\": \"Folder for monthly client-report automations against AIP-protected workbooks.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6130001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/task.yaml index 4484164057..15f4ce71d1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-card-sensitivity-label-rejection/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/README.md new file mode 100644 index 0000000000..9d02f77875 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/README.md @@ -0,0 +1,63 @@ +# Excel Delete Range — Filter Misalignment (Silent Corruption) + +This scenario reproduces a Delete Range failure where the activity +itself succeeds but silently deletes hidden rows because of an +active AutoFilter, causing a downstream `Throw` to fire when an +expected customer record is missing from the post-delete data. The +job ends with: + +``` +System.Exception: Customer CUST-9001 missing from post-delete data +``` + +The visible runtime fault is the Throw — but that's the symptom. +The root is Delete Range's coordinate-mode behavior against a +filtered sheet. + +## What this scenario uncovers + +**Root Cause:** The workflow's Delete Range targets +`Customers!A10:D20` with `ShiftCells=True ShiftOption=ShiftCellsUp`, +but the workbook has an active AutoFilter on column `Status` hiding +rows where `Status='Archived'`. `CUST-9001` sits on a hidden row +inside the deletion target. Delete Range operates on cell +coordinates, not visible rows, so `CUST-9001` is silently removed +along with the intentional test-data rows. The downstream +`For Each Row` + `Throw` post-condition check fires because +`CUST-9001` is not in the post-delete DataTable. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/delete-range-failures.md` +(the "Range misalignment over filtered data" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelCustomerProcess` project — `Use Excel File` → `Read Range` → `Delete Range A10:D20 ShiftCells=True` → `Read Range` → `For Each Row` → conditional `Throw` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (the visible fault is the +Throw — NOT a Delete Range exception) → `jobs logs` (decisive: the +AutoFilter Trace line + Delete Range "completed successfully" Trace +line are the smoking gun) → workflow source review (confirms the +Read → Delete → Read → ForEach → Throw sequence and that Delete +Range targeted a coordinate range, not a content-based selection) → +conclude branch 5. + +> **Note on the agent's challenge.** The exception text says +> "Customer CUST-9001 missing." A naive agent might match a customer +> data-quality playbook or blame the source data. The correct +> conclusion requires connecting the AutoFilter state from the logs +> to the silent Delete Range corruption — reasoning backward from +> the Throw through the workflow sequence and the log timeline. + +> **Note on fixtures.** Synthetic. The customer ID, sheet name, +> and visible/hidden row counts are placeholders. The test grades +> whether the agent traces the failure back to Delete Range + +> AutoFilter (not just observes the Throw) and recommends a viable +> fix (remove the filter, refactor to DataTable, or delete by key). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/07d8f224aa.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/07d8f224aa.json deleted file mode 100644 index e09d671a62..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/07d8f224aa.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-23T08:00:01.300Z", - "EndTime": "2026-05-23T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9dd33333-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Exception: Customer CUST-9001 missing from post-delete data\r\n at System.Activities.Statements.Throw.Execute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-23T08:00:01.005Z", - "ReleaseName": "ExcelCustomerProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680401, - "OrganizationUnitId": 6040001, - "FolderKey": "f0044444-3333-4444-5555-666677778888", - "Reference": "CustomerCleanupA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-23T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "Exception", - "ActivityName": "Throw_1", - "ActivityDisplayName": "Throw: missing customer", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680399001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/2e39ca11ce.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/2e39ca11ce.json deleted file mode 100644 index 87cc4a73a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/2e39ca11ce.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0044444-3333-4444-5555-666677778888", - "DisplayName": "CustomerOps", - "FullyQualifiedName": "CustomerOps", - "Description": "Folder for customer-list maintenance automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6040001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/46b20bd4b3.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/46b20bd4b3.json deleted file mode 100644 index 49a8e98619..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/46b20bd4b3.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-dd3333cc-001", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-23T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-002", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-23T08:00:01.418Z", - "Message": "[ExcelCustomerProcess] Removing legacy test customers (rows 10-20) from C:\\Robot\\Data\\customer-list.xlsx", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-003", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-23T08:00:01.510Z", - "Message": "Use Excel File: customer-list.xlsx — workbook opened (OpenXML provider)", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-004", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-23T08:00:01.610Z", - "Message": "Use Excel File: customer-list.xlsx — workbook has active AutoFilter on sheet 'Customers' column 'Status' (hiding rows where Status='Archived')", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-005", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-23T08:00:01.810Z", - "Message": "Read Range: pre-delete snapshot (Customers!A1:Z100) — returned 47 rows (filtered view: 22 visible, 25 hidden including rows 15-22)", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-006", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-23T08:00:02.110Z", - "Message": "Delete Range: drop legacy customers (Customers!A10:D20 ShiftCells=True ShiftOption=ShiftCellsUp) — completed successfully (44 cells deleted across 11 rows, 0 errors)", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-007", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-23T08:00:02.410Z", - "Message": "Read Range: post-delete snapshot (Customers!A1:Z100) — returned 36 rows", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-008", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-23T08:00:02.610Z", - "Message": "For Each Row: scan for CUST-9001 — 36 iterations, CUST-9001 not encountered (found=False)", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd3333cc-009", - "JobKey": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-23T08:00:02.802Z", - "Message": "Throw: missing customer — System.Exception: Customer CUST-9001 missing from post-delete data", - "ProcessName": "ExcelCustomerProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/8a1045b8bc.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/8a1045b8bc.json deleted file mode 100644 index 006a3d5be0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/8a1045b8bc.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-23T08:00:01.300Z", - "EndTime": "2026-05-23T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelCustomerProcess", - "Source": "Schedule", - "Reference": "CustomerCleanupA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0044444-3333-4444-5555-666677778888", - "Info": "System.Exception: Customer CUST-9001 missing from post-delete data", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/94a675ff10.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/94a675ff10.json deleted file mode 100644 index e875ee29d1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/94a675ff10.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-23T08:00:01.005Z"}, - {"Key": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-23T08:00:01.300Z"}, - {"Key": "dd333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-23T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/README.md deleted file mode 100644 index 43fc672673..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# Excel Delete Range — Filter Misalignment (Silent Corruption) - -This scenario reproduces a Delete Range failure where the activity -itself succeeds but silently deletes hidden rows because of an -active AutoFilter, causing a downstream `Throw` to fire when an -expected customer record is missing from the post-delete data. The -job ends with: - -``` -System.Exception: Customer CUST-9001 missing from post-delete data -``` - -The visible runtime fault is the Throw — but that's the symptom. -The root is Delete Range's coordinate-mode behavior against a -filtered sheet. - -## What this scenario uncovers - -**Root Cause:** The workflow's Delete Range targets -`Customers!A10:D20` with `ShiftCells=True ShiftOption=ShiftCellsUp`, -but the workbook has an active AutoFilter on column `Status` hiding -rows where `Status='Archived'`. `CUST-9001` sits on a hidden row -inside the deletion target. Delete Range operates on cell -coordinates, not visible rows, so `CUST-9001` is silently removed -along with the intentional test-data rows. The downstream -`For Each Row` + `Throw` post-condition check fires because -`CUST-9001` is not in the post-delete DataTable. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/delete-range-failures.md` -(the "Range misalignment over filtered data" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelCustomerProcess` project — `Use Excel File` → `Read Range` → `Delete Range A10:D20 ShiftCells=True` → `Read Range` → `For Each Row` → conditional `Throw` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture the active AutoFilter, the successful Delete Range completion, and the For Each Row "CUST-9001 not encountered" result that triggers the Throw | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (the visible fault is the -Throw — NOT a Delete Range exception) → `jobs logs` (decisive: the -AutoFilter Trace line + Delete Range "completed successfully" Trace -line are the smoking gun) → workflow source review (confirms the -Read → Delete → Read → ForEach → Throw sequence and that Delete -Range targeted a coordinate range, not a content-based selection) → -conclude branch 5. - -> **Note on the agent's challenge.** The exception text says -> "Customer CUST-9001 missing." A naive agent might match a customer -> data-quality playbook or blame the source data. The correct -> conclusion requires connecting the AutoFilter state from the logs -> to the silent Delete Range corruption — reasoning backward from -> the Throw through the workflow sequence and the log timeline. - -> **Note on fixtures.** Synthetic. The customer ID, sheet name, -> and visible/hidden row counts are placeholders. The test grades -> whether the agent traces the failure back to Delete Range + -> AutoFilter (not just observes the Throw) and recommends a viable -> fix (remove the filter, refactor to DataTable, or delete by key). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/manifest.json deleted file mode 100644 index 9622ec1371..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Delete Range customer-list cleanup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "2e39ca11ce.json"}, - {"match": "or folders list --output json", "file": "2e39ca11ce.json"}, - {"match": "or jobs list --folder-key \"f0044444-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "8a1045b8bc.json"}, - {"match": "or jobs list --folder-key f0044444-3333-4444-5555-666677778888 --state Faulted --output json", "file": "8a1045b8bc.json"}, - {"match": "or jobs get \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "07d8f224aa.json"}, - {"match": "or jobs get dd333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "07d8f224aa.json"}, - {"match": "or jobs history \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "94a675ff10.json"}, - {"match": "or jobs history dd333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "94a675ff10.json"}, - {"match": "or jobs logs \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "46b20bd4b3.json"}, - {"match": "or jobs logs dd333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "46b20bd4b3.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/uip-fixture.json new file mode 100644 index 0000000000..39c596f726 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0044444-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-23T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-23T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelCustomerProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"CustomerCleanupA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0044444-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.Exception: Customer CUST-9001 missing from post-delete data\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "dd333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-23T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-23T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9dd33333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Exception: Customer CUST-9001 missing from post-delete data\\r\\n at System.Activities.Statements.Throw.Execute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-23T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelCustomerProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680401,\r\n \"OrganizationUnitId\": 6040001,\r\n \"FolderKey\": \"f0044444-3333-4444-5555-666677778888\",\r\n \"Reference\": \"CustomerCleanupA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-23T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"Exception\",\r\n \"ActivityName\": \"Throw_1\",\r\n \"ActivityDisplayName\": \"Throw: missing customer\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680399001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "dd333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-23T08:00:01.005Z\"},\r\n {\"Key\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-23T08:00:01.300Z\"},\r\n {\"Key\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-23T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dd333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-dd3333cc-001\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-23T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-002\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-23T08:00:01.418Z\",\r\n \"Message\": \"[ExcelCustomerProcess] Removing legacy test customers (rows 10-20) from C:\\\\Robot\\\\Data\\\\customer-list.xlsx\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-003\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: customer-list.xlsx — workbook opened (OpenXML provider)\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-004\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T08:00:01.610Z\",\r\n \"Message\": \"Use Excel File: customer-list.xlsx — workbook has active AutoFilter on sheet 'Customers' column 'Status' (hiding rows where Status='Archived')\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-005\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T08:00:01.810Z\",\r\n \"Message\": \"Read Range: pre-delete snapshot (Customers!A1:Z100) — returned 47 rows (filtered view: 22 visible, 25 hidden including rows 15-22)\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-006\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T08:00:02.110Z\",\r\n \"Message\": \"Delete Range: drop legacy customers (Customers!A10:D20 ShiftCells=True ShiftOption=ShiftCellsUp) — completed successfully (44 cells deleted across 11 rows, 0 errors)\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-007\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T08:00:02.410Z\",\r\n \"Message\": \"Read Range: post-delete snapshot (Customers!A1:Z100) — returned 36 rows\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-008\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T08:00:02.610Z\",\r\n \"Message\": \"For Each Row: scan for CUST-9001 — 36 iterations, CUST-9001 not encountered (found=False)\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd3333cc-009\",\r\n \"JobKey\": \"dd333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-23T08:00:02.802Z\",\r\n \"Message\": \"Throw: missing customer — System.Exception: Customer CUST-9001 missing from post-delete data\",\r\n \"ProcessName\": \"ExcelCustomerProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0044444-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"CustomerOps\",\r\n \"FullyQualifiedName\": \"CustomerOps\",\r\n \"Description\": \"Folder for customer-list maintenance automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6040001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/task.yaml index f902428a4a..70fdf1d2a9 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-filter-misalignment/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/README.md new file mode 100644 index 0000000000..3bc5a3e36f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/README.md @@ -0,0 +1,47 @@ +# Excel Delete Range — Invalid Range Syntax + +This scenario reproduces a Delete Range failure where the activity's +`Range` property is an expression that resolves to an invalid A1 +string at runtime. The job ends with: + +``` +System.ArgumentException: The range is invalid +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's Delete Range activity is configured +with `Range = "A1:B" + lastRow.ToString()`. The preceding Read Range +returned 0 rows from a header-only workbook, so +`lastRow = dtCleanup.Rows.Count = 0`, and the expression evaluated to +`"A1:B0"`. Row index 0 is not a legal A1 address; the Modern +`DeleteRangeX` activity's range validator rejected the string with +an `ArgumentException` before any provider call was made. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/delete-range-failures.md` +(the "Invalid range syntax or empty Range property" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelCleanupProcess` project — `Use Excel File` → `Read Range` → `Assign lastRow` → `Delete Range` with `Range = "A1:B" + lastRow.ToString()` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (ArgumentException +"The range is invalid" on Delete Range) → `jobs logs` (resolved +Range expression "A1:B0" with lastRow=0 is the smoking gun) → +workflow source review (confirms `Range` is a `lastRow`-dependent +expression) → conclude branch 2. + +> **Note on fixtures.** Synthetic. The workbook path, row counts, +> and timing values are placeholders. The test grades whether the +> agent identifies the resolved range value as invalid A1 syntax +> and recommends either guarding the expression, switching to +> Clear Range, or using a literal range — not just observing the +> exception. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/683e47118b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/683e47118b.json deleted file mode 100644 index b59e51d4b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/683e47118b.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022222-3333-4444-5555-666677778888", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/6847a787a0.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/6847a787a0.json deleted file mode 100644 index b14fdda495..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/6847a787a0.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-21T08:00:01.300Z", - "EndTime": "2026-05-21T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9dd11111-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.ArgumentException: The range is invalid\r\n at UiPath.Excel.Activities.Business.DeleteRangeX.ValidateRange(String range)\r\n at UiPath.Excel.Activities.Business.DeleteRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-21T08:00:01.005Z", - "ReleaseName": "ExcelCleanupProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680201, - "OrganizationUnitId": 6020001, - "FolderKey": "f0022222-3333-4444-5555-666677778888", - "Reference": "DailyCleanupA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "ArgumentException", - "ActivityName": "DeleteRange_1", - "ActivityDisplayName": "Delete Range: clear stale block", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680199001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/7e405fc6d4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/7e405fc6d4.json deleted file mode 100644 index 71df2c0c27..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/7e405fc6d4.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-dd1111cc-001", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-21T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd1111cc-002", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-21T08:00:01.418Z", - "Message": "[ExcelCleanupProcess] Starting cleanup — wiping stale data from C:\\Robot\\Data\\sales-cleanup.xlsx", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd1111cc-003", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-21T08:00:01.510Z", - "Message": "Use Excel File: sales-cleanup.xlsx — workbook opened (OpenXML provider)", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd1111cc-004", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-21T08:00:01.910Z", - "Message": "Read Range: scan used data (Sales!A1:Z1000) — returned 0 rows (header-only workbook)", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd1111cc-005", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-21T08:00:02.110Z", - "Message": "Assign: lastRow = dtCleanup.Rows.Count → 0", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd1111cc-006", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-21T08:00:02.510Z", - "Message": "Delete Range: clear stale block — resolved Range expression \"A1:B\" + lastRow.ToString() to \"A1:B0\"", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd1111cc-007", - "JobKey": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-21T08:00:02.802Z", - "Message": "Delete Range: clear stale block — System.ArgumentException: The range is invalid", - "ProcessName": "ExcelCleanupProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/9d2f4acf36.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/9d2f4acf36.json deleted file mode 100644 index 8cec00a904..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/9d2f4acf36.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-21T08:00:01.005Z"}, - {"Key": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-21T08:00:01.300Z"}, - {"Key": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-21T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/README.md deleted file mode 100644 index e6f90acf2c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Excel Delete Range — Invalid Range Syntax - -This scenario reproduces a Delete Range failure where the activity's -`Range` property is an expression that resolves to an invalid A1 -string at runtime. The job ends with: - -``` -System.ArgumentException: The range is invalid -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's Delete Range activity is configured -with `Range = "A1:B" + lastRow.ToString()`. The preceding Read Range -returned 0 rows from a header-only workbook, so -`lastRow = dtCleanup.Rows.Count = 0`, and the expression evaluated to -`"A1:B0"`. Row index 0 is not a legal A1 address; the Modern -`DeleteRangeX` activity's range validator rejected the string with -an `ArgumentException` before any provider call was made. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/delete-range-failures.md` -(the "Invalid range syntax or empty Range property" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelCleanupProcess` project — `Use Excel File` → `Read Range` → `Assign lastRow` → `Delete Range` with `Range = "A1:B" + lastRow.ToString()` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entry shows the resolved expression evaluated to `"A1:B0"` immediately before the ArgumentException | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (ArgumentException -"The range is invalid" on Delete Range) → `jobs logs` (resolved -Range expression "A1:B0" with lastRow=0 is the smoking gun) → -workflow source review (confirms `Range` is a `lastRow`-dependent -expression) → conclude branch 2. - -> **Note on fixtures.** Synthetic. The workbook path, row counts, -> and timing values are placeholders. The test grades whether the -> agent identifies the resolved range value as invalid A1 syntax -> and recommends either guarding the expression, switching to -> Clear Range, or using a literal range — not just observing the -> exception. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/b0f683e76c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/b0f683e76c.json deleted file mode 100644 index c51433b2de..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/b0f683e76c.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "dd111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-21T08:00:01.300Z", - "EndTime": "2026-05-21T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelCleanupProcess", - "Source": "Schedule", - "Reference": "DailyCleanupA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022222-3333-4444-5555-666677778888", - "Info": "System.ArgumentException: The range is invalid", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/manifest.json deleted file mode 100644 index 2fa6d93166..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Delete Range sales-cleanup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "683e47118b.json"}, - {"match": "or folders list --output json", "file": "683e47118b.json"}, - {"match": "or jobs list --folder-key \"f0022222-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "b0f683e76c.json"}, - {"match": "or jobs list --folder-key f0022222-3333-4444-5555-666677778888 --state Faulted --output json", "file": "b0f683e76c.json"}, - {"match": "or jobs get \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "6847a787a0.json"}, - {"match": "or jobs get dd111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "6847a787a0.json"}, - {"match": "or jobs history \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "9d2f4acf36.json"}, - {"match": "or jobs history dd111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "9d2f4acf36.json"}, - {"match": "or jobs logs \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "7e405fc6d4.json"}, - {"match": "or jobs logs dd111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "7e405fc6d4.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/uip-fixture.json new file mode 100644 index 0000000000..638d07a05e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0022222-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-21T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-21T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelCleanupProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailyCleanupA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022222-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.ArgumentException: The range is invalid\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "dd111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-21T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-21T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9dd11111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.ArgumentException: The range is invalid\\r\\n at UiPath.Excel.Activities.Business.DeleteRangeX.ValidateRange(String range)\\r\\n at UiPath.Excel.Activities.Business.DeleteRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-21T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelCleanupProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680201,\r\n \"OrganizationUnitId\": 6020001,\r\n \"FolderKey\": \"f0022222-3333-4444-5555-666677778888\",\r\n \"Reference\": \"DailyCleanupA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"ArgumentException\",\r\n \"ActivityName\": \"DeleteRange_1\",\r\n \"ActivityDisplayName\": \"Delete Range: clear stale block\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680199001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "dd111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-21T08:00:01.005Z\"},\r\n {\"Key\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-21T08:00:01.300Z\"},\r\n {\"Key\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-21T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dd111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-dd1111cc-001\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-21T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd1111cc-002\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-21T08:00:01.418Z\",\r\n \"Message\": \"[ExcelCleanupProcess] Starting cleanup — wiping stale data from C:\\\\Robot\\\\Data\\\\sales-cleanup.xlsx\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd1111cc-003\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-21T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-cleanup.xlsx — workbook opened (OpenXML provider)\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd1111cc-004\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-21T08:00:01.910Z\",\r\n \"Message\": \"Read Range: scan used data (Sales!A1:Z1000) — returned 0 rows (header-only workbook)\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd1111cc-005\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-21T08:00:02.110Z\",\r\n \"Message\": \"Assign: lastRow = dtCleanup.Rows.Count → 0\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd1111cc-006\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-21T08:00:02.510Z\",\r\n \"Message\": \"Delete Range: clear stale block — resolved Range expression \\\"A1:B\\\" + lastRow.ToString() to \\\"A1:B0\\\"\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd1111cc-007\",\r\n \"JobKey\": \"dd111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-21T08:00:02.802Z\",\r\n \"Message\": \"Delete Range: clear stale block — System.ArgumentException: The range is invalid\",\r\n \"ProcessName\": \"ExcelCleanupProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022222-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/task.yaml index 6705b5ef6e..d96a912b6f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-invalid-syntax/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/README.md new file mode 100644 index 0000000000..33008fb229 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/README.md @@ -0,0 +1,45 @@ +# Excel Delete Range — Shift Conflict with Merged Region + +This scenario reproduces a Delete Range failure where `ShiftCells=True` +with `ShiftOption=ShiftUp` collides with a merged region anchored on +the bottom edge of the deletion target. The job ends with: + +``` +System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's Delete Range activity targets +`Q1!A3:C5` with `ShiftCells=True ShiftOption=ShiftUp`. Sheet `Q1` +has a merged region `D5:D7` whose anchor sits on the bottom edge of +the deletion target. Excel's `Range.Delete xlShiftUp` cannot move +rows 6-7 into row 5 without breaking the merge, so Excel COM rejects +the call with the generic `0x800A03EC` Application-defined error. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/delete-range-failures.md` +(the "ShiftCells / ShiftOption conflict" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelQuarterlyProcess` project — Classic `Excel Application Scope` → `Delete Range` with Range=A3:C5, ShiftCells=True, ShiftOption=ShiftUp | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (COMException 0x800A03EC +on Delete Range) → `jobs logs` (merged-region scan + ShiftCells +config + COM dispatch failure) → workflow source review (confirms +ShiftCells=True ShiftOption=ShiftUp on A3:C5) → conclude branch 3. + +> **Note on fixtures.** Synthetic. The merged region's exact +> coordinates and HRESULT detail are placeholders. The test grades +> whether the agent connects the ShiftCells/ShiftOption configuration +> to the merged-region collision and recommends a viable fix (disable +> shift, switch to EntireRow, switch to Clear Range, or restructure +> the merge). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/096bfec52c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/096bfec52c.json deleted file mode 100644 index 2888424077..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/096bfec52c.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-22T08:00:01.300Z", - "EndTime": "2026-05-22T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelQuarterlyProcess", - "Source": "Schedule", - "Reference": "QuarterlyTrimA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0033333-3333-4444-5555-666677778888", - "Info": "System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/4b25ecc978.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/4b25ecc978.json deleted file mode 100644 index 0e55f3d702..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/4b25ecc978.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0033333-3333-4444-5555-666677778888", - "DisplayName": "Quarterly", - "FullyQualifiedName": "Quarterly", - "Description": "Folder for quarterly reporting automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6030001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/51f311c01d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/51f311c01d.json deleted file mode 100644 index 8159a4addb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/51f311c01d.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-22T08:00:01.005Z"}, - {"Key": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-22T08:00:01.300Z"}, - {"Key": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-22T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/README.md deleted file mode 100644 index 34060c712f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Excel Delete Range — Shift Conflict with Merged Region - -This scenario reproduces a Delete Range failure where `ShiftCells=True` -with `ShiftOption=ShiftUp` collides with a merged region anchored on -the bottom edge of the deletion target. The job ends with: - -``` -System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's Delete Range activity targets -`Q1!A3:C5` with `ShiftCells=True ShiftOption=ShiftUp`. Sheet `Q1` -has a merged region `D5:D7` whose anchor sits on the bottom edge of -the deletion target. Excel's `Range.Delete xlShiftUp` cannot move -rows 6-7 into row 5 without breaking the merge, so Excel COM rejects -the call with the generic `0x800A03EC` Application-defined error. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/delete-range-failures.md` -(the "ShiftCells / ShiftOption conflict" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelQuarterlyProcess` project — Classic `Excel Application Scope` → `Delete Range` with Range=A3:C5, ShiftCells=True, ShiftOption=ShiftUp | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entry captures both the resolved Delete Range config AND the merged-region scan that detected `D5:D7` on the bottom edge | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (COMException 0x800A03EC -on Delete Range) → `jobs logs` (merged-region scan + ShiftCells -config + COM dispatch failure) → workflow source review (confirms -ShiftCells=True ShiftOption=ShiftUp on A3:C5) → conclude branch 3. - -> **Note on fixtures.** Synthetic. The merged region's exact -> coordinates and HRESULT detail are placeholders. The test grades -> whether the agent connects the ShiftCells/ShiftOption configuration -> to the merged-region collision and recommends a viable fix (disable -> shift, switch to EntireRow, switch to Clear Range, or restructure -> the merge). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/bde18e0f9c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/bde18e0f9c.json deleted file mode 100644 index f937990a20..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/bde18e0f9c.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-dd2222cc-001", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-22T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd2222cc-002", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-22T08:00:01.418Z", - "Message": "[ExcelQuarterlyProcess] Removing stale block A3:C5 from C:\\Robot\\Data\\quarterly-report.xlsx", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd2222cc-003", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-22T08:00:01.510Z", - "Message": "Excel Application Scope: quarterly-report.xlsx — workbook opened (Visible=False, AutoSave=True)", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd2222cc-004", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-22T08:00:02.010Z", - "Message": "Delete Range: drop stale block — resolved config Sheet='Q1' Range='A3:C5' ShiftCells=True ShiftOption=ShiftUp", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd2222cc-005", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-22T08:00:02.210Z", - "Message": "Delete Range: drop stale block — sheet 'Q1' merged regions adjacent to target: ['D5:D7'] (bottom edge of A3:C5 abuts merge anchor at D5)", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd2222cc-006", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-22T08:00:02.510Z", - "Message": "Delete Range: drop stale block — dispatching Excel COM Range.Delete(xlShiftUp) on Q1!A3:C5", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd2222cc-007", - "JobKey": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-22T08:00:02.802Z", - "Message": "Delete Range: drop stale block — System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.", - "ProcessName": "ExcelQuarterlyProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/d899dedfcb.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/d899dedfcb.json deleted file mode 100644 index 126fd38ecf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/d899dedfcb.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-22T08:00:01.300Z", - "EndTime": "2026-05-22T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9dd22222-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\r\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\r\n at Microsoft.Office.Interop.Excel.Range.Delete(Object Shift)\r\n at UiPath.Excel.Activities.DeleteRange.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-22T08:00:01.005Z", - "ReleaseName": "ExcelQuarterlyProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680301, - "OrganizationUnitId": 6030001, - "FolderKey": "f0033333-3333-4444-5555-666677778888", - "Reference": "QuarterlyTrimA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-22T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "ActivityName": "DeleteRange_1", - "ActivityDisplayName": "Delete Range: drop stale block", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "dd222222-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680299001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/manifest.json deleted file mode 100644 index 9ac895c8ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Delete Range quarterly-report trim scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "4b25ecc978.json"}, - {"match": "or folders list --output json", "file": "4b25ecc978.json"}, - {"match": "or jobs list --folder-key \"f0033333-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "096bfec52c.json"}, - {"match": "or jobs list --folder-key f0033333-3333-4444-5555-666677778888 --state Faulted --output json", "file": "096bfec52c.json"}, - {"match": "or jobs get \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "d899dedfcb.json"}, - {"match": "or jobs get dd222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "d899dedfcb.json"}, - {"match": "or jobs history \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "51f311c01d.json"}, - {"match": "or jobs history dd222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "51f311c01d.json"}, - {"match": "or jobs logs \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "bde18e0f9c.json"}, - {"match": "or jobs logs dd222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "bde18e0f9c.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/uip-fixture.json new file mode 100644 index 0000000000..68660793db --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0033333-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-22T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-22T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelQuarterlyProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"QuarterlyTrimA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0033333-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "dd222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-22T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-22T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9dd22222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\\r\\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\\r\\n at Microsoft.Office.Interop.Excel.Range.Delete(Object Shift)\\r\\n at UiPath.Excel.Activities.DeleteRange.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-22T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelQuarterlyProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680301,\r\n \"OrganizationUnitId\": 6030001,\r\n \"FolderKey\": \"f0033333-3333-4444-5555-666677778888\",\r\n \"Reference\": \"QuarterlyTrimA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-22T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"ActivityName\": \"DeleteRange_1\",\r\n \"ActivityDisplayName\": \"Delete Range: drop stale block\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680299001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "dd222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-22T08:00:01.005Z\"},\r\n {\"Key\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-22T08:00:01.300Z\"},\r\n {\"Key\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-22T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dd222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-dd2222cc-001\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-22T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd2222cc-002\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-22T08:00:01.418Z\",\r\n \"Message\": \"[ExcelQuarterlyProcess] Removing stale block A3:C5 from C:\\\\Robot\\\\Data\\\\quarterly-report.xlsx\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd2222cc-003\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T08:00:01.510Z\",\r\n \"Message\": \"Excel Application Scope: quarterly-report.xlsx — workbook opened (Visible=False, AutoSave=True)\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd2222cc-004\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T08:00:02.010Z\",\r\n \"Message\": \"Delete Range: drop stale block — resolved config Sheet='Q1' Range='A3:C5' ShiftCells=True ShiftOption=ShiftUp\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd2222cc-005\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T08:00:02.210Z\",\r\n \"Message\": \"Delete Range: drop stale block — sheet 'Q1' merged regions adjacent to target: ['D5:D7'] (bottom edge of A3:C5 abuts merge anchor at D5)\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd2222cc-006\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T08:00:02.510Z\",\r\n \"Message\": \"Delete Range: drop stale block — dispatching Excel COM Range.Delete(xlShiftUp) on Q1!A3:C5\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd2222cc-007\",\r\n \"JobKey\": \"dd222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-22T08:00:02.802Z\",\r\n \"Message\": \"Delete Range: drop stale block — System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\",\r\n \"ProcessName\": \"ExcelQuarterlyProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0033333-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"Quarterly\",\r\n \"FullyQualifiedName\": \"Quarterly\",\r\n \"Description\": \"Folder for quarterly reporting automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6030001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/task.yaml index d7ae30172e..f087de76fe 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-delete-range-shift-conflict/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/README.md new file mode 100644 index 0000000000..c7576849af --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/README.md @@ -0,0 +1,48 @@ +# Invoke VBA Failure - Code File Missing Sub/Function Wrapper + +This scenario reproduces a runtime `Invoke VBA` failure caused by the +external code file at `CodeFilePath` containing **bare VBA statements at +the top level** (no `Sub` or `Function` wrapper). The VBA compiler +rejects the injected module before any macro logic runs. + +## What this scenario uncovers + +**Root Cause:** `macro.txt` in the project contains bare executable +statements (`ActiveWorkbook.RefreshAll`, `Application.Calculate`) at the +module level, with no `Sub RefreshSheets()` / `End Sub` block around +them. `Invoke VBA` injects the file as a module body; the VBA compiler +expects only procedure declarations at the module level, so the +injection compile-fails with `Expected: Sub, Function, Property, or +Type`. + +This maps to: +`references/activity-packages/excel-activities/playbooks/invoke-vba-code-file-path.md` +(specifically the "Code not wrapped in a Sub/Function" branch). + +The user is framed as **on the project source**, so the agent has both +`Main.xaml` (the workflow + `CodeFilePath` reference) and `macro.txt` +(the broken source) directly Read-able. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/Main.xaml` + `process/macro.txt` | hand-authored UiPath project with an Excel Process Scope + Invoke VBA referencing a `CodeFilePath` whose contents are deliberately missing the `Sub` wrapper | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `invoke-vba-code-file-path.md` +- Agent named the missing `Sub`/`Function` wrapper in `macro.txt` and + recommended wrapping the contents in a `Sub RefreshSheets()` block + matching the activity's `EntryMethodName` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/070192a730.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/070192a730.json deleted file mode 100644 index 89741efe0c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/070192a730.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-13T10:42:18.250Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T10:42:18.811Z", - "Message": "[Excel Process Scope] Opening workbook reports\\WeeklyKPI.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T10:42:21.020Z", - "Message": "[Use Excel File] Workbook opened successfully" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T10:42:22.041Z", - "Message": "[Invoke VBA] Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T10:42:22.088Z", - "Message": "Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T10:42:22.105Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/1bf0c1366c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/1bf0c1366c.json deleted file mode 100644 index 709c9c6259..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/1bf0c1366c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T10:42:22.041Z", - "Message": "[Invoke VBA] Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T10:42:22.088Z", - "Message": "Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/2440c6971a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/2440c6971a.json deleted file mode 100644 index 11150668c8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/2440c6971a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60", - "StartTime": "2026-05-13T10:42:18.220Z", - "EndTime": "2026-05-13T10:42:22.110Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f9a1b2c3-d4e5-46f7-8a9b-0c1d2e3f4a5b", - "Info": "Invoke VBA: Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\r\n\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type", - "CreationTime": "2026-05-13T10:42:18.005Z", - "ReleaseName": "ReportRefresher", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6018245, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T10:42:22.110Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60", - "Id": 682315502 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/5ce3bd35b4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/5ce3bd35b4.json deleted file mode 100644 index e24caa7f86..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/5ce3bd35b4.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60", - "State": "Faulted", - "ProcessName": "ReportRefresher", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T10:42:18.220Z", - "EndTime": "2026-05-13T10:42:22.110Z", - "CreationTime": "2026-05-13T10:42:18.005Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/README.md deleted file mode 100644 index dda04b8962..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Invoke VBA Failure - Code File Missing Sub/Function Wrapper - -This scenario reproduces a runtime `Invoke VBA` failure caused by the -external code file at `CodeFilePath` containing **bare VBA statements at -the top level** (no `Sub` or `Function` wrapper). The VBA compiler -rejects the injected module before any macro logic runs. - -## What this scenario uncovers - -**Root Cause:** `macro.txt` in the project contains bare executable -statements (`ActiveWorkbook.RefreshAll`, `Application.Calculate`) at the -module level, with no `Sub RefreshSheets()` / `End Sub` block around -them. `Invoke VBA` injects the file as a module body; the VBA compiler -expects only procedure declarations at the module level, so the -injection compile-fails with `Expected: Sub, Function, Property, or -Type`. - -This maps to: -`references/activity-packages/excel-activities/playbooks/invoke-vba-code-file-path.md` -(specifically the "Code not wrapped in a Sub/Function" branch). - -The user is framed as **on the project source**, so the agent has both -`Main.xaml` (the workflow + `CodeFilePath` reference) and `macro.txt` -(the broken source) directly Read-able. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/Main.xaml` + `process/macro.txt` | hand-authored UiPath project with an Excel Process Scope + Invoke VBA referencing a `CodeFilePath` whose contents are deliberately missing the `Sub` wrapper | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `invoke-vba-code-file-path.md` -- Agent named the missing `Sub`/`Function` wrapper in `macro.txt` and - recommended wrapping the contents in a `Sub RefreshSheets()` block - matching the activity's `EntryMethodName` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/manifest.json deleted file mode 100644 index 7afb4953c4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Invoke VBA WeeklyKPI refresh scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "5ce3bd35b4.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "5ce3bd35b4.json" - }, - { - "match": "or jobs get \"e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60\" --output json", - "file": "2440c6971a.json" - }, - { - "match": "or jobs get e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60 --output json", - "file": "2440c6971a.json" - }, - { - "match": "or jobs logs \"e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60\" --level Error --output json", - "file": "1bf0c1366c.json" - }, - { - "match": "or jobs logs e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60 --level Error --output json", - "file": "1bf0c1366c.json" - }, - { - "match": "or jobs logs \"e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60\" --output json", - "file": "070192a730.json" - }, - { - "match": "or jobs logs e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60 --output json", - "file": "070192a730.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/uip-fixture.json new file mode 100644 index 0000000000..3495669f4f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportRefresher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T10:42:18.220Z\",\r\n \"EndTime\": \"2026-05-13T10:42:22.110Z\",\r\n \"CreationTime\": \"2026-05-13T10:42:18.005Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T10:42:22.041Z\",\r\n \"Message\": \"[Invoke VBA] Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T10:42:22.088Z\",\r\n \"Message\": \"Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60\",\r\n \"StartTime\": \"2026-05-13T10:42:18.220Z\",\r\n \"EndTime\": \"2026-05-13T10:42:22.110Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f9a1b2c3-d4e5-46f7-8a9b-0c1d2e3f4a5b\",\r\n \"Info\": \"Invoke VBA: Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\",\r\n \"CreationTime\": \"2026-05-13T10:42:18.005Z\",\r\n \"ReleaseName\": \"ReportRefresher\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6018245,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T10:42:22.110Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60\",\r\n \"Id\": 682315502\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e8c7d6f5-4a3b-4c2d-9e1f-0a2b3c4d5e60", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T10:42:18.250Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T10:42:18.811Z\",\r\n \"Message\": \"[Excel Process Scope] Opening workbook reports\\\\WeeklyKPI.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T10:42:21.020Z\",\r\n \"Message\": \"[Use Excel File] Workbook opened successfully\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T10:42:22.041Z\",\r\n \"Message\": \"[Invoke VBA] Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T10:42:22.088Z\",\r\n \"Message\": \"Cannot run the macro 'RefreshSheets'. Compile error: Expected: Sub, Function, Property, or Type\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T10:42:22.105Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/task.yaml index 50e9ab911c..32f630183a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-code-file/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/README.md new file mode 100644 index 0000000000..1078b0f8d9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/README.md @@ -0,0 +1,53 @@ +# Invoke VBA Failure - COM Interop HRESULT 0x80010100 + +This scenario reproduces a runtime `Invoke VBA` failure caused by a COM +interop instability: a hidden modal dialog on Excel (Protected View +bar from Mark-of-the-Web) blocks the COM call into the workbook, while +the surrounding Excel Process Scope ran with `ShowExcel=False`. The +COM layer surfaces the wedge as +`The system call failed. (Exception from HRESULT: 0x80010100 +(RPC_E_SYS_CALL_FAILED))`. + +## What this scenario uncovers + +**Root Cause:** The Excel Process Scope in `Main.xaml` runs with +`ShowExcel="False"`. When the workbook opens, a hidden Protected View +bar (from a Mark-of-the-Web on the downloaded `.xlsx`) blocks every +COM call into the workbook. The Invoke VBA activity's call into the +workbook's VBProject times out and the COM layer raises +`0x80010100 RPC_E_SYS_CALL_FAILED`. The same workbook + macro pair +runs cleanly once the Protected View bar is dismissed and the file is +unblocked. + +This maps to: +`references/activity-packages/excel-activities/playbooks/invoke-vba-com-interop-failure.md` +(specifically the "modal dialog was blocking" branch). + +The user is framed as **off-host**, so the correct agent behavior is +to recommend the host-side check list (Visible=True, dialog +inspection, orphaned EXCEL.EXE, Office install) rather than try +commands itself. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `ShowExcel="False"` on the Excel Process Scope | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `invoke-vba-com-interop-failure.md` +- Agent recommended setting `ShowExcel`/`Visible` to True on the Excel + Process Scope to surface the hidden dialog, and listed the other + host-side checks (Protected View / Mark-of-the-Web, orphaned + EXCEL.EXE, multi-Office hosts) without fabricating host commands it + could not run + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/0455da7851.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/0455da7851.json deleted file mode 100644 index 8af576d64f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/0455da7851.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-16T07:00:12.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-16T07:00:12.722Z", - "Message": "[Excel Process Scope] Opening workbook reports\\DailyDigest.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-16T07:00:12.901Z", - "Message": "[Excel Process Scope] Cleared 0 residual EXCEL.EXE processes before start (KillExcelProcessesEachIteration=True)" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-16T07:00:18.014Z", - "Message": "[Use Excel File] Workbook opened successfully" - }, - { - "Level": "Warn", - "TimeStamp": "2026-05-16T07:00:21.502Z", - "Message": "[Invoke VBA] COM call to Workbook.VBProject is not responding; the Excel session is not pumping window messages - behaviour consistent with a modal dialog awaiting input on a non-visible Excel instance" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-16T07:00:24.241Z", - "Message": "[Invoke VBA] The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-16T07:00:24.292Z", - "Message": "The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-16T07:00:24.305Z", - "Message": "Execution ended in: 00:00:12" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/31b5b62aa6.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/31b5b62aa6.json deleted file mode 100644 index 05653ac79a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/31b5b62aa6.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912", - "State": "Faulted", - "ProcessName": "DailyDigestRunner", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-16T07:00:12.110Z", - "EndTime": "2026-05-16T07:00:24.310Z", - "CreationTime": "2026-05-16T07:00:11.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/68e1080a12.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/68e1080a12.json deleted file mode 100644 index 52086dd089..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/68e1080a12.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912", - "StartTime": "2026-05-16T07:00:12.110Z", - "EndTime": "2026-05-16T07:00:24.310Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "cd4e5f60-7182-4839-4051-6f7a8b90c1d2", - "Info": "Invoke VBA: The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\r\n\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))", - "CreationTime": "2026-05-16T07:00:11.880Z", - "ReleaseName": "DailyDigestRunner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6018311, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-16T07:00:24.310Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912", - "Id": 682317602 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/README.md deleted file mode 100644 index aa12e77296..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# Invoke VBA Failure - COM Interop HRESULT 0x80010100 - -This scenario reproduces a runtime `Invoke VBA` failure caused by a COM -interop instability: a hidden modal dialog on Excel (Protected View -bar from Mark-of-the-Web) blocks the COM call into the workbook, while -the surrounding Excel Process Scope ran with `ShowExcel=False`. The -COM layer surfaces the wedge as -`The system call failed. (Exception from HRESULT: 0x80010100 -(RPC_E_SYS_CALL_FAILED))`. - -## What this scenario uncovers - -**Root Cause:** The Excel Process Scope in `Main.xaml` runs with -`ShowExcel="False"`. When the workbook opens, a hidden Protected View -bar (from a Mark-of-the-Web on the downloaded `.xlsx`) blocks every -COM call into the workbook. The Invoke VBA activity's call into the -workbook's VBProject times out and the COM layer raises -`0x80010100 RPC_E_SYS_CALL_FAILED`. The same workbook + macro pair -runs cleanly once the Protected View bar is dismissed and the file is -unblocked. - -This maps to: -`references/activity-packages/excel-activities/playbooks/invoke-vba-com-interop-failure.md` -(specifically the "modal dialog was blocking" branch). - -The user is framed as **off-host**, so the correct agent behavior is -to recommend the host-side check list (Visible=True, dialog -inspection, orphaned EXCEL.EXE, Office install) rather than try -commands itself. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `ShowExcel="False"` on the Excel Process Scope | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `invoke-vba-com-interop-failure.md` -- Agent recommended setting `ShowExcel`/`Visible` to True on the Excel - Process Scope to surface the hidden dialog, and listed the other - host-side checks (Protected View / Mark-of-the-Web, orphaned - EXCEL.EXE, multi-Office hosts) without fabricating host commands it - could not run diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/afe6979ec7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/afe6979ec7.json deleted file mode 100644 index efe4695ed5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/afe6979ec7.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-16T07:00:24.241Z", - "Message": "[Invoke VBA] The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-16T07:00:24.292Z", - "Message": "The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/manifest.json deleted file mode 100644 index 4f178f0624..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Invoke VBA DailyDigest build scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "31b5b62aa6.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "31b5b62aa6.json" - }, - { - "match": "or jobs get \"b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912\" --output json", - "file": "68e1080a12.json" - }, - { - "match": "or jobs get b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912 --output json", - "file": "68e1080a12.json" - }, - { - "match": "or jobs logs \"b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912\" --level Error --output json", - "file": "afe6979ec7.json" - }, - { - "match": "or jobs logs b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912 --level Error --output json", - "file": "afe6979ec7.json" - }, - { - "match": "or jobs logs \"b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912\" --output json", - "file": "0455da7851.json" - }, - { - "match": "or jobs logs b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912 --output json", - "file": "0455da7851.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/uip-fixture.json new file mode 100644 index 0000000000..584fb357a0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyDigestRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-16T07:00:12.110Z\",\r\n \"EndTime\": \"2026-05-16T07:00:24.310Z\",\r\n \"CreationTime\": \"2026-05-16T07:00:11.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-16T07:00:24.241Z\",\r\n \"Message\": \"[Invoke VBA] The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-16T07:00:24.292Z\",\r\n \"Message\": \"The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912\",\r\n \"StartTime\": \"2026-05-16T07:00:12.110Z\",\r\n \"EndTime\": \"2026-05-16T07:00:24.310Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"cd4e5f60-7182-4839-4051-6f7a8b90c1d2\",\r\n \"Info\": \"Invoke VBA: The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\",\r\n \"CreationTime\": \"2026-05-16T07:00:11.880Z\",\r\n \"ReleaseName\": \"DailyDigestRunner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6018311,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-16T07:00:24.310Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912\",\r\n \"Id\": 682317602\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b3c4d5e6-f7a8-4901-2bc3-d4e5f6a78912", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-16T07:00:12.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-16T07:00:12.722Z\",\r\n \"Message\": \"[Excel Process Scope] Opening workbook reports\\\\DailyDigest.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-16T07:00:12.901Z\",\r\n \"Message\": \"[Excel Process Scope] Cleared 0 residual EXCEL.EXE processes before start (KillExcelProcessesEachIteration=True)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-16T07:00:18.014Z\",\r\n \"Message\": \"[Use Excel File] Workbook opened successfully\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-05-16T07:00:21.502Z\",\r\n \"Message\": \"[Invoke VBA] COM call to Workbook.VBProject is not responding; the Excel session is not pumping window messages - behaviour consistent with a modal dialog awaiting input on a non-visible Excel instance\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-16T07:00:24.241Z\",\r\n \"Message\": \"[Invoke VBA] The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-16T07:00:24.292Z\",\r\n \"Message\": \"The system call failed. (Exception from HRESULT: 0x80010100 (RPC_E_SYS_CALL_FAILED))\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-16T07:00:24.305Z\",\r\n \"Message\": \"Execution ended in: 00:00:12\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/task.yaml index 9080b2b06f..32028ca8c2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-com-interop/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/README.md new file mode 100644 index 0000000000..1b2ce1d4ea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/README.md @@ -0,0 +1,18 @@ +# Invoke VBA EntryMethodName - de-DE localized canary (translate-forcing) + +Locale-robustness canary #2: unlike the classic-movefile canary, this scenario's playbook +(`invoke-vba-entry-method-name.md`) is routable ONLY by message signatures ("Cannot run the +macro" / "Sub or Function not defined") - there is no language-invariant exception class or +error code for it in the playbook corpus. The fixtures' only exception class (COMException) +appears in an unrelated Word playbook whose preconditions do not fit. + +All Office/VBA error messages in the fixtures are localized to authentic de-DE wording. +Routing therefore succeeds only via the skill's translate-before-grep rule: read the German +message, translate to canonical English, grep the playbooks. CLI-only (no source snapshot), so +the localized runtime evidence cannot be bypassed. + +Grading: identical contract to the English variant (same root cause, same fix). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/3950fd9841.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/3950fd9841.json deleted file mode 100644 index e37693b98b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/3950fd9841.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", - "State": "Faulted", - "ProcessName": "InvoiceImporter", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T08:05:11.110Z", - "EndTime": "2026-05-14T08:05:16.420Z", - "CreationTime": "2026-05-14T08:05:10.885Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/46ff0750bc.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/46ff0750bc.json deleted file mode 100644 index 57e7f48de1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/46ff0750bc.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.341Z", - "Message": "[Invoke VBA] Das Makro \"ProcessInvoces\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.398Z", - "Message": "Das Makro \"ProcessInvoces\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert.\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/6c5a0e2eba.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/6c5a0e2eba.json deleted file mode 100644 index dd68c6700b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/6c5a0e2eba.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", - "StartTime": "2026-05-14T08:05:11.110Z", - "EndTime": "2026-05-14T08:05:16.420Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ab1c2d3e-4f5a-46b7-8c9d-0e1f2a3b4c5d", - "Info": "Invoke VBA: Das Makro \"ProcessInvoces\" kann nicht ausgeführt werden. The macro may not be available in this workbook or all macros may be disabled\r\n\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Das Makro \"ProcessInvoces\" kann nicht ausgeführt werden. The macro may not be available in this workbook or all macros may be disabled", - "CreationTime": "2026-05-14T08:05:10.885Z", - "ReleaseName": "InvoiceImporter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6018266, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-14T08:05:16.420Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", - "Id": 682316108 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/README.md deleted file mode 100644 index d5cce8cbc6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Invoke VBA EntryMethodName - de-DE localized canary (translate-forcing) - -Locale-robustness canary #2: unlike the classic-movefile canary, this scenario's playbook -(`invoke-vba-entry-method-name.md`) is routable ONLY by message signatures ("Cannot run the -macro" / "Sub or Function not defined") - there is no language-invariant exception class or -error code for it in the playbook corpus. The fixtures' only exception class (COMException) -appears in an unrelated Word playbook whose preconditions do not fit. - -All Office/VBA error messages in the fixtures are localized to authentic de-DE wording. -Routing therefore succeeds only via the skill's translate-before-grep rule: read the German -message, translate to canonical English, grep the playbooks. CLI-only (no source snapshot), so -the localized runtime evidence cannot be bypassed. - -Grading: identical contract to the English variant (same root cause, same fix). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/fca66714f0.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/fca66714f0.json deleted file mode 100644 index 5dae441aa8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/fca66714f0.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:11.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:11.722Z", - "Message": "[Excel Process Scope] Opening workbook data\\Invoices.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:15.014Z", - "Message": "[Use Excel File] Workbook opened successfully" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.341Z", - "Message": "[Invoke VBA] Das Makro \"ProcessInvoces\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.398Z", - "Message": "Das Makro \"ProcessInvoces\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert.\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:16.415Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/manifest.json deleted file mode 100644 index f3b457f3fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Invoke VBA invoice-processing scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "3950fd9841.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "3950fd9841.json" - }, - { - "match": "or jobs get \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\" --output json", - "file": "6c5a0e2eba.json" - }, - { - "match": "or jobs get f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019 --output json", - "file": "6c5a0e2eba.json" - }, - { - "match": "or jobs logs \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\" --level Error --output json", - "file": "46ff0750bc.json" - }, - { - "match": "or jobs logs f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019 --level Error --output json", - "file": "46ff0750bc.json" - }, - { - "match": "or jobs logs \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\" --output json", - "file": "fca66714f0.json" - }, - { - "match": "or jobs logs f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019 --output json", - "file": "fca66714f0.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/uip-fixture.json new file mode 100644 index 0000000000..401a5eb0f9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceImporter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T08:05:11.110Z\",\r\n \"EndTime\": \"2026-05-14T08:05:16.420Z\",\r\n \"CreationTime\": \"2026-05-14T08:05:10.885Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.341Z\",\r\n \"Message\": \"[Invoke VBA] Das Makro \\\"ProcessInvoces\\\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.398Z\",\r\n \"Message\": \"Das Makro \\\"ProcessInvoces\\\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert.\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\",\r\n \"StartTime\": \"2026-05-14T08:05:11.110Z\",\r\n \"EndTime\": \"2026-05-14T08:05:16.420Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ab1c2d3e-4f5a-46b7-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"Invoke VBA: Das Makro \\\"ProcessInvoces\\\" kann nicht ausgeführt werden. The macro may not be available in this workbook or all macros may be disabled\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Das Makro \\\"ProcessInvoces\\\" kann nicht ausgeführt werden. The macro may not be available in this workbook or all macros may be disabled\",\r\n \"CreationTime\": \"2026-05-14T08:05:10.885Z\",\r\n \"ReleaseName\": \"InvoiceImporter\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6018266,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-14T08:05:16.420Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\",\r\n \"Id\": 682316108\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:11.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:11.722Z\",\r\n \"Message\": \"[Excel Process Scope] Opening workbook data\\\\Invoices.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:15.014Z\",\r\n \"Message\": \"[Use Excel File] Workbook opened successfully\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.341Z\",\r\n \"Message\": \"[Invoke VBA] Das Makro \\\"ProcessInvoces\\\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.398Z\",\r\n \"Message\": \"Das Makro \\\"ProcessInvoces\\\" kann nicht ausgeführt werden. Das Makro ist möglicherweise nicht in dieser Arbeitsmappe vorhanden, oder alle Makros wurden deaktiviert.\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.415Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/task.yaml index dea2ba3ab9..f46fe0477f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name-de/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/README.md new file mode 100644 index 0000000000..31b5ff603a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/README.md @@ -0,0 +1,46 @@ +# Invoke VBA Failure - Entry Method Name Typo + +This scenario reproduces a runtime `Invoke VBA` failure caused by an +`EntryMethodName` typo in the workflow XAML. `Application.Run` cannot +resolve the typo'd name against the correctly-spelled `Sub` in the +external code file. + +## What this scenario uncovers + +**Root Cause:** The `Invoke VBA` activity in `Main.xaml` sets +`EntryMethodName="ProcessInvoces"` (missing an "i"). The external code +file `macro.txt` declares `Sub ProcessInvoices()` (correctly spelled). +`Application.Run` looks up the name verbatim against the injected +module and finds no match, so the activity faults with `Cannot run the +macro 'ProcessInvoces'. The macro may not be available in this +workbook`. + +This maps to: +`references/activity-packages/excel-activities/playbooks/invoke-vba-entry-method-name.md` +(specifically the "typo or whitespace mismatch" branch). + +The user is framed as **on the project source**, so the agent has both +`Main.xaml` (with the typo'd `EntryMethodName`) and `macro.txt` (with +the correctly-spelled Sub) directly Read-able. This mirrors the existing +`getasset-name-mismatch` scenario's typo-near-miss pattern. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/Main.xaml` + `process/macro.txt` | hand-authored UiPath project with a typo'd `EntryMethodName` and a correctly-spelled Sub | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `invoke-vba-entry-method-name.md` +- Agent named the typo explicitly (`ProcessInvoces` vs + `ProcessInvoices`), not just "macro name doesn't match" + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/3950fd9841.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/3950fd9841.json deleted file mode 100644 index e37693b98b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/3950fd9841.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", - "State": "Faulted", - "ProcessName": "InvoiceImporter", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T08:05:11.110Z", - "EndTime": "2026-05-14T08:05:16.420Z", - "CreationTime": "2026-05-14T08:05:10.885Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/46ff0750bc.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/46ff0750bc.json deleted file mode 100644 index b71f895919..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/46ff0750bc.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.341Z", - "Message": "[Invoke VBA] Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.398Z", - "Message": "Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled.\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/6c5a0e2eba.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/6c5a0e2eba.json deleted file mode 100644 index a1194a27f0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/6c5a0e2eba.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", - "StartTime": "2026-05-14T08:05:11.110Z", - "EndTime": "2026-05-14T08:05:16.420Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ab1c2d3e-4f5a-46b7-8c9d-0e1f2a3b4c5d", - "Info": "Invoke VBA: Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled\r\n\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled", - "CreationTime": "2026-05-14T08:05:10.885Z", - "ReleaseName": "InvoiceImporter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6018266, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-14T08:05:16.420Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", - "Id": 682316108 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/README.md deleted file mode 100644 index 068c55f920..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Invoke VBA Failure - Entry Method Name Typo - -This scenario reproduces a runtime `Invoke VBA` failure caused by an -`EntryMethodName` typo in the workflow XAML. `Application.Run` cannot -resolve the typo'd name against the correctly-spelled `Sub` in the -external code file. - -## What this scenario uncovers - -**Root Cause:** The `Invoke VBA` activity in `Main.xaml` sets -`EntryMethodName="ProcessInvoces"` (missing an "i"). The external code -file `macro.txt` declares `Sub ProcessInvoices()` (correctly spelled). -`Application.Run` looks up the name verbatim against the injected -module and finds no match, so the activity faults with `Cannot run the -macro 'ProcessInvoces'. The macro may not be available in this -workbook`. - -This maps to: -`references/activity-packages/excel-activities/playbooks/invoke-vba-entry-method-name.md` -(specifically the "typo or whitespace mismatch" branch). - -The user is framed as **on the project source**, so the agent has both -`Main.xaml` (with the typo'd `EntryMethodName`) and `macro.txt` (with -the correctly-spelled Sub) directly Read-able. This mirrors the existing -`getasset-name-mismatch` scenario's typo-near-miss pattern. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/Main.xaml` + `process/macro.txt` | hand-authored UiPath project with a typo'd `EntryMethodName` and a correctly-spelled Sub | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `invoke-vba-entry-method-name.md` -- Agent named the typo explicitly (`ProcessInvoces` vs - `ProcessInvoices`), not just "macro name doesn't match" diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/fca66714f0.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/fca66714f0.json deleted file mode 100644 index 96fcf06dff..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/fca66714f0.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:11.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:11.722Z", - "Message": "[Excel Process Scope] Opening workbook data\\Invoices.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:15.014Z", - "Message": "[Use Excel File] Workbook opened successfully" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.341Z", - "Message": "[Invoke VBA] Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-14T08:05:16.398Z", - "Message": "Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled.\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-14T08:05:16.415Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/manifest.json deleted file mode 100644 index f3b457f3fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Invoke VBA invoice-processing scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "3950fd9841.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "3950fd9841.json" - }, - { - "match": "or jobs get \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\" --output json", - "file": "6c5a0e2eba.json" - }, - { - "match": "or jobs get f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019 --output json", - "file": "6c5a0e2eba.json" - }, - { - "match": "or jobs logs \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\" --level Error --output json", - "file": "46ff0750bc.json" - }, - { - "match": "or jobs logs f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019 --level Error --output json", - "file": "46ff0750bc.json" - }, - { - "match": "or jobs logs \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\" --output json", - "file": "fca66714f0.json" - }, - { - "match": "or jobs logs f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019 --output json", - "file": "fca66714f0.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/uip-fixture.json new file mode 100644 index 0000000000..36fdddec3c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceImporter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T08:05:11.110Z\",\r\n \"EndTime\": \"2026-05-14T08:05:16.420Z\",\r\n \"CreationTime\": \"2026-05-14T08:05:10.885Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.341Z\",\r\n \"Message\": \"[Invoke VBA] Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.398Z\",\r\n \"Message\": \"Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled.\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\",\r\n \"StartTime\": \"2026-05-14T08:05:11.110Z\",\r\n \"EndTime\": \"2026-05-14T08:05:16.420Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ab1c2d3e-4f5a-46b7-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"Invoke VBA: Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled\",\r\n \"CreationTime\": \"2026-05-14T08:05:10.885Z\",\r\n \"ReleaseName\": \"InvoiceImporter\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6018266,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-14T08:05:16.420Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019\",\r\n \"Id\": 682316108\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7e6d5c4-3b2a-4019-8f7e-6d5c4b3a2019", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:11.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:11.722Z\",\r\n \"Message\": \"[Excel Process Scope] Opening workbook data\\\\Invoices.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:15.014Z\",\r\n \"Message\": \"[Use Excel File] Workbook opened successfully\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.341Z\",\r\n \"Message\": \"[Invoke VBA] Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.398Z\",\r\n \"Message\": \"Cannot run the macro 'ProcessInvoces'. The macro may not be available in this workbook or all macros may be disabled.\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-14T08:05:16.415Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/task.yaml index 44447fda96..31caad04a0 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-method-name/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/README.md new file mode 100644 index 0000000000..0415d8786a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/README.md @@ -0,0 +1,45 @@ +# Invoke VBA Failure - Parameter Arity Mismatch + +This scenario reproduces a runtime `Invoke VBA` failure caused by a +mismatch between the macro's signature in `macro.txt` and the +`EntryMethodParameters` array passed by `Main.xaml`. The Sub requires +two arguments but the workflow passes only one. + +## What this scenario uncovers + +**Root Cause:** `macro.txt` declares +`Sub PostRow(amount As Double, vendor As String)` with two required +parameters. The Invoke VBA activity in `Main.xaml` passes +`new Object[] { 125.50 }` - a one-element array. `Application.Run` +rejects the arity mismatch with `Wrong number of arguments or invalid +property assignment`. + +This maps to: +`references/activity-packages/excel-activities/playbooks/invoke-vba-parameter-formatting.md` +(specifically the "arity mismatch" branch). + +The user is framed as **on the project source**, so the agent has both +`Main.xaml` (the parameter expression) and `macro.txt` (the Sub +signature) directly Read-able. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/Main.xaml` + `process/macro.txt` | hand-authored UiPath project with a 1-arg parameter array against a 2-arg Sub signature | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `invoke-vba-parameter-formatting.md` +- Agent named the arity mismatch (2 required parameters in the Sub vs + 1-element array in the XAML) and recommended passing a 2-element + array + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/3558635336.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/3558635336.json deleted file mode 100644 index 724e77c03c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/3558635336.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708", - "State": "Faulted", - "ProcessName": "LedgerPoster", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T11:18:30.110Z", - "EndTime": "2026-05-15T11:18:34.715Z", - "CreationTime": "2026-05-15T11:18:29.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/6e73041592.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/6e73041592.json deleted file mode 100644 index feb48c8bae..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/6e73041592.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-15T11:18:34.642Z", - "Message": "[Invoke VBA] Wrong number of arguments or invalid property assignment" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-15T11:18:34.695Z", - "Message": "Wrong number of arguments or invalid property assignment\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/README.md deleted file mode 100644 index dcd27fffdf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Invoke VBA Failure - Parameter Arity Mismatch - -This scenario reproduces a runtime `Invoke VBA` failure caused by a -mismatch between the macro's signature in `macro.txt` and the -`EntryMethodParameters` array passed by `Main.xaml`. The Sub requires -two arguments but the workflow passes only one. - -## What this scenario uncovers - -**Root Cause:** `macro.txt` declares -`Sub PostRow(amount As Double, vendor As String)` with two required -parameters. The Invoke VBA activity in `Main.xaml` passes -`new Object[] { 125.50 }` - a one-element array. `Application.Run` -rejects the arity mismatch with `Wrong number of arguments or invalid -property assignment`. - -This maps to: -`references/activity-packages/excel-activities/playbooks/invoke-vba-parameter-formatting.md` -(specifically the "arity mismatch" branch). - -The user is framed as **on the project source**, so the agent has both -`Main.xaml` (the parameter expression) and `macro.txt` (the Sub -signature) directly Read-able. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/Main.xaml` + `process/macro.txt` | hand-authored UiPath project with a 1-arg parameter array against a 2-arg Sub signature | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `invoke-vba-parameter-formatting.md` -- Agent named the arity mismatch (2 required parameters in the Sub vs - 1-element array in the XAML) and recommended passing a 2-element - array diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/bed086c6c1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/bed086c6c1.json deleted file mode 100644 index 9f695c0a36..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/bed086c6c1.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-15T11:18:30.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-15T11:18:30.722Z", - "Message": "[Excel Process Scope] Opening workbook data\\Ledger.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-15T11:18:33.311Z", - "Message": "[Use Excel File] Workbook opened successfully" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-15T11:18:34.642Z", - "Message": "[Invoke VBA] Wrong number of arguments or invalid property assignment" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-15T11:18:34.695Z", - "Message": "Wrong number of arguments or invalid property assignment\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-15T11:18:34.710Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/ff41a8d6d2.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/ff41a8d6d2.json deleted file mode 100644 index 42bc0b98dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/ff41a8d6d2.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708", - "StartTime": "2026-05-15T11:18:30.110Z", - "EndTime": "2026-05-15T11:18:34.715Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "bc2d3e4f-5061-4728-9304-5d6e7f80a1b2", - "Info": "Invoke VBA: Wrong number of arguments or invalid property assignment\r\n\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Wrong number of arguments or invalid property assignment", - "CreationTime": "2026-05-15T11:18:29.880Z", - "ReleaseName": "LedgerPoster", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6018288, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-15T11:18:34.715Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708", - "Id": 682316845 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/manifest.json deleted file mode 100644 index 4aa865f9a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Invoke VBA ledger-posting scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "3558635336.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "3558635336.json" - }, - { - "match": "or jobs get \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708\" --output json", - "file": "ff41a8d6d2.json" - }, - { - "match": "or jobs get a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708 --output json", - "file": "ff41a8d6d2.json" - }, - { - "match": "or jobs logs \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708\" --level Error --output json", - "file": "6e73041592.json" - }, - { - "match": "or jobs logs a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708 --level Error --output json", - "file": "6e73041592.json" - }, - { - "match": "or jobs logs \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708\" --output json", - "file": "bed086c6c1.json" - }, - { - "match": "or jobs logs a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708 --output json", - "file": "bed086c6c1.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/uip-fixture.json new file mode 100644 index 0000000000..4ffccefc54 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LedgerPoster\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T11:18:30.110Z\",\r\n \"EndTime\": \"2026-05-15T11:18:34.715Z\",\r\n \"CreationTime\": \"2026-05-15T11:18:29.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-15T11:18:34.642Z\",\r\n \"Message\": \"[Invoke VBA] Wrong number of arguments or invalid property assignment\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-15T11:18:34.695Z\",\r\n \"Message\": \"Wrong number of arguments or invalid property assignment\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708\",\r\n \"StartTime\": \"2026-05-15T11:18:30.110Z\",\r\n \"EndTime\": \"2026-05-15T11:18:34.715Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"bc2d3e4f-5061-4728-9304-5d6e7f80a1b2\",\r\n \"Info\": \"Invoke VBA: Wrong number of arguments or invalid property assignment\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Wrong number of arguments or invalid property assignment\",\r\n \"CreationTime\": \"2026-05-15T11:18:29.880Z\",\r\n \"ReleaseName\": \"LedgerPoster\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6018288,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-15T11:18:34.715Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708\",\r\n \"Id\": 682316845\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6708", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-15T11:18:30.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-15T11:18:30.722Z\",\r\n \"Message\": \"[Excel Process Scope] Opening workbook data\\\\Ledger.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-15T11:18:33.311Z\",\r\n \"Message\": \"[Use Excel File] Workbook opened successfully\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-15T11:18:34.642Z\",\r\n \"Message\": \"[Invoke VBA] Wrong number of arguments or invalid property assignment\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-15T11:18:34.695Z\",\r\n \"Message\": \"Wrong number of arguments or invalid property assignment\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-15T11:18:34.710Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/task.yaml index 2bec1e3bee..1aa70012ad 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-params/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/README.md new file mode 100644 index 0000000000..f57afaa731 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/README.md @@ -0,0 +1,48 @@ +# Invoke VBA Failure - Trust Access to VBA Project Denied + +This scenario reproduces a runtime `Invoke VBA` failure caused by Excel's +**"Trust access to the VBA project object model"** Trust Center setting +being disabled on the robot machine. The activity throws +`Programmatic access to Visual Basic Project is not trusted` synchronously +when it attempts to inject the macro module into the workbook. + +## What this scenario uncovers + +**Root Cause:** Excel's "Trust access to the VBA project object model" +setting is disabled under the Windows user the robot runs as. The +`Invoke VBA` activity in `Main.xaml` cannot call +`Workbook.VBProject.VBComponents.Add` to inject the macro module, so the +job faults before any macro logic executes. + +This maps to: +`references/activity-packages/excel-activities/playbooks/invoke-vba-trust-access.md` + +The user is framed as **off-host**, so the correct agent behavior is to +recommend the Trust Center toggle the next time the user is at the +machine - not to try host-side commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a minimal Excel Process Scope containing an Invoke VBA activity, plus the external `macro.txt` it references | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. Before treating this test's score as +> a regression signal, replay against a real failed job and regenerate +> via `_shared/scripts/generate_scenario.py` - see +> `tests/tasks/uipath-troubleshoot/CLAUDE.md`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `invoke-vba-trust-access.md` +- Agent recommended enabling the Trust Center toggle on the robot + machine, without fabricating host-side actions it could not perform + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/05cc4b9617.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/05cc4b9617.json deleted file mode 100644 index 702333bfe6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/05cc4b9617.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70", - "State": "Faulted", - "ProcessName": "InvoiceMacroRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T09:14:02.110Z", - "EndTime": "2026-05-12T09:14:05.310Z", - "CreationTime": "2026-05-12T09:14:01.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/658e904d39.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/658e904d39.json deleted file mode 100644 index 6acfb67034..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/658e904d39.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T09:14:05.241Z", - "Message": "[Invoke VBA] Programmatic access to Visual Basic Project is not trusted (HRESULT: 0x800AC472)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T09:14:05.288Z", - "Message": "Programmatic access to Visual Basic Project is not trusted\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/6f0197671a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/6f0197671a.json deleted file mode 100644 index 1f01e7b37c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/6f0197671a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70", - "StartTime": "2026-05-12T09:14:02.110Z", - "EndTime": "2026-05-12T09:14:05.310Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ec70a2b8-5d3f-4e1c-9b8a-3c5d7e9f0a12", - "Info": "Invoke VBA: Programmatic access to Visual Basic Project is not trusted\r\n\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Programmatic access to Visual Basic Project is not trusted\r\n at UiPath.Excel.Activities.Business.InvokeVBAX.", - "CreationTime": "2026-05-12T09:14:01.880Z", - "ReleaseName": "InvoiceMacroRunner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6018221, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T09:14:05.310Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70", - "Id": 682310445 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/8c0893cc79.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/8c0893cc79.json deleted file mode 100644 index 70aeb4f146..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/8c0893cc79.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-12T09:14:02.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T09:14:02.612Z", - "Message": "[Excel Process Scope] Opening workbook reports\\Q3Pivot.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T09:14:04.220Z", - "Message": "[Use Excel File] Workbook opened successfully" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T09:14:05.241Z", - "Message": "[Invoke VBA] Programmatic access to Visual Basic Project is not trusted (HRESULT: 0x800AC472)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T09:14:05.288Z", - "Message": "Programmatic access to Visual Basic Project is not trusted\r\nin Main.xaml\r\n at InvokeVBAX \"Invoke VBA\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T09:14:05.305Z", - "Message": "Execution ended in: 00:00:03" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/README.md deleted file mode 100644 index 9583012312..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Invoke VBA Failure - Trust Access to VBA Project Denied - -This scenario reproduces a runtime `Invoke VBA` failure caused by Excel's -**"Trust access to the VBA project object model"** Trust Center setting -being disabled on the robot machine. The activity throws -`Programmatic access to Visual Basic Project is not trusted` synchronously -when it attempts to inject the macro module into the workbook. - -## What this scenario uncovers - -**Root Cause:** Excel's "Trust access to the VBA project object model" -setting is disabled under the Windows user the robot runs as. The -`Invoke VBA` activity in `Main.xaml` cannot call -`Workbook.VBProject.VBComponents.Add` to inject the macro module, so the -job faults before any macro logic executes. - -This maps to: -`references/activity-packages/excel-activities/playbooks/invoke-vba-trust-access.md` - -The user is framed as **off-host**, so the correct agent behavior is to -recommend the Trust Center toggle the next time the user is at the -machine - not to try host-side commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | hand-authored UiPath project with a minimal Excel Process Scope containing an Invoke VBA activity, plus the external `macro.txt` it references | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. Before treating this test's score as -> a regression signal, replay against a real failed job and regenerate -> via `_shared/scripts/generate_scenario.py` - see -> `tests/tasks/uipath-troubleshoot/CLAUDE.md`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `invoke-vba-trust-access.md` -- Agent recommended enabling the Trust Center toggle on the robot - machine, without fabricating host-side actions it could not perform diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/manifest.json deleted file mode 100644 index 4677158316..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Invoke VBA Q3Pivot refresh scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "05cc4b9617.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "05cc4b9617.json" - }, - { - "match": "or jobs get \"d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70\" --output json", - "file": "6f0197671a.json" - }, - { - "match": "or jobs get d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70 --output json", - "file": "6f0197671a.json" - }, - { - "match": "or jobs logs \"d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70\" --level Error --output json", - "file": "658e904d39.json" - }, - { - "match": "or jobs logs d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70 --level Error --output json", - "file": "658e904d39.json" - }, - { - "match": "or jobs logs \"d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70\" --output json", - "file": "8c0893cc79.json" - }, - { - "match": "or jobs logs d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70 --output json", - "file": "8c0893cc79.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/uip-fixture.json new file mode 100644 index 0000000000..0027a3ad0d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceMacroRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T09:14:02.110Z\",\r\n \"EndTime\": \"2026-05-12T09:14:05.310Z\",\r\n \"CreationTime\": \"2026-05-12T09:14:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T09:14:05.241Z\",\r\n \"Message\": \"[Invoke VBA] Programmatic access to Visual Basic Project is not trusted (HRESULT: 0x800AC472)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T09:14:05.288Z\",\r\n \"Message\": \"Programmatic access to Visual Basic Project is not trusted\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70\",\r\n \"StartTime\": \"2026-05-12T09:14:02.110Z\",\r\n \"EndTime\": \"2026-05-12T09:14:05.310Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ec70a2b8-5d3f-4e1c-9b8a-3c5d7e9f0a12\",\r\n \"Info\": \"Invoke VBA: Programmatic access to Visual Basic Project is not trusted\\r\\n\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Programmatic access to Visual Basic Project is not trusted\\r\\n at UiPath.Excel.Activities.Business.InvokeVBAX.\",\r\n \"CreationTime\": \"2026-05-12T09:14:01.880Z\",\r\n \"ReleaseName\": \"InvoiceMacroRunner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6018221,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-12T09:14:05.310Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70\",\r\n \"Id\": 682310445\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d3f1a2b8-4e6c-49d7-8f1a-2b3c4d5e6f70", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T09:14:02.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T09:14:02.612Z\",\r\n \"Message\": \"[Excel Process Scope] Opening workbook reports\\\\Q3Pivot.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T09:14:04.220Z\",\r\n \"Message\": \"[Use Excel File] Workbook opened successfully\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T09:14:05.241Z\",\r\n \"Message\": \"[Invoke VBA] Programmatic access to Visual Basic Project is not trusted (HRESULT: 0x800AC472)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T09:14:05.288Z\",\r\n \"Message\": \"Programmatic access to Visual Basic Project is not trusted\\r\\nin Main.xaml\\r\\n at InvokeVBAX \\\"Invoke VBA\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T09:14:05.305Z\",\r\n \"Message\": \"Execution ended in: 00:00:03\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/task.yaml index db631d13fb..7932c49353 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-invokevba-trust-access/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/README.md new file mode 100644 index 0000000000..71c6990959 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/README.md @@ -0,0 +1,44 @@ +# Lookup Range Failure - Value Not Found Behind an Active Filter + +This scenario reproduces a `Lookup Range` "value not found" failure where +the searched value is physically present in the sheet but hidden by an +**active AutoFilter** that the workflow applied earlier. Lookup Range +searches only visible cells, so the filtered-out row is invisible to it. + +## What this scenario uncovers + +**Root Cause:** `Main.xaml` applies a Filter activity ("Filter Status = +Active") to the `Prices` sheet before the `Lookup Range` runs. The target +`SKU-8842` has Status "Discontinued", so the active filter hides its row. +Lookup Range returns empty, and the workflow's guard throws +`SKU-8842 not found in price list`. The value is genuinely present - it +is filtered out, not absent. + +This maps to: +`references/activity-packages/excel-activities/playbooks/lookup-range-active-filters.md` + +The Filter-before-Lookup ordering is **discoverable in Main.xaml**, so +this is a local investigation with a definite answer (no need to read the +binary workbook). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/Main.xaml` | Use Excel File -> Filter (Status = Active) -> Lookup Range (SKU-8842) -> Throw-if-empty | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `lookup-range-active-filters.md` +- Agent identified the active filter (applied before the lookup) as why a + present value is not matched, and recommended clearing/resetting the + filter before the lookup + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/6bed206584.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/6bed206584.json deleted file mode 100644 index c588a7948e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/6bed206584.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01", - "StartTime": "2026-05-27T09:12:40.420Z", - "EndTime": "2026-05-27T09:12:46.330Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "e7f8090a-1b2c-4d3e-4f5a-6b7c8d9e0f12", - "Info": "Throw SKU Not Found: SKU-8842 not found in price list\r\n\r\nin Main.xaml\r\n at Throw \"Throw SKU Not Found\"\r\n at If \"Guard empty result\"\r\n at Sequence \"Process Price List\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.BusinessRuleException: SKU-8842 not found in price list", - "CreationTime": "2026-05-27T09:12:40.190Z", - "ReleaseName": "PriceUpdater", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020188, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T09:12:46.330Z", - "JobError": {}, - "ErrorCode": "BusinessRuleException", - "TraceId": "d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01", - "Id": 682404519 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/README.md deleted file mode 100644 index 9e39ea56a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Lookup Range Failure - Value Not Found Behind an Active Filter - -This scenario reproduces a `Lookup Range` "value not found" failure where -the searched value is physically present in the sheet but hidden by an -**active AutoFilter** that the workflow applied earlier. Lookup Range -searches only visible cells, so the filtered-out row is invisible to it. - -## What this scenario uncovers - -**Root Cause:** `Main.xaml` applies a Filter activity ("Filter Status = -Active") to the `Prices` sheet before the `Lookup Range` runs. The target -`SKU-8842` has Status "Discontinued", so the active filter hides its row. -Lookup Range returns empty, and the workflow's guard throws -`SKU-8842 not found in price list`. The value is genuinely present - it -is filtered out, not absent. - -This maps to: -`references/activity-packages/excel-activities/playbooks/lookup-range-active-filters.md` - -The Filter-before-Lookup ordering is **discoverable in Main.xaml**, so -this is a local investigation with a definite answer (no need to read the -binary workbook). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/Main.xaml` | Use Excel File -> Filter (Status = Active) -> Lookup Range (SKU-8842) -> Throw-if-empty | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `lookup-range-active-filters.md` -- Agent identified the active filter (applied before the lookup) as why a - present value is not matched, and recommended clearing/resetting the - filter before the lookup diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/c797d9ede2.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/c797d9ede2.json deleted file mode 100644 index 984cd6a222..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/c797d9ede2.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-27T09:12:40.450Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T09:12:42.870Z", - "Message": "[Use Excel File] Opened workbook data\\PriceList.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T09:12:44.110Z", - "Message": "[Filter Status = Active] Applied filter on sheet 'Prices' column 'Status' = 'Active'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-27T09:12:45.880Z", - "Message": "[Lookup Range] Searched sheet 'Prices' for 'SKU-8842'; Result = '' (no match in visible range)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T09:12:46.262Z", - "Message": "[Throw SKU Not Found] SKU-8842 not found in price list" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T09:12:46.305Z", - "Message": "SKU-8842 not found in price list\r\nin Main.xaml\r\n at Throw \"Throw SKU Not Found\"\r\n at If \"Guard empty result\"\r\n at Sequence \"Process Price List\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T09:12:46.325Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/d3150e31e3.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/d3150e31e3.json deleted file mode 100644 index 6d0747d52f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/d3150e31e3.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-27T09:12:46.262Z", - "Message": "[Throw SKU Not Found] SKU-8842 not found in price list" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T09:12:46.305Z", - "Message": "SKU-8842 not found in price list\r\nin Main.xaml\r\n at Throw \"Throw SKU Not Found\"\r\n at If \"Guard empty result\"\r\n at Sequence \"Process Price List\"\r\n at ExcelApplicationCard \"Use Excel File\"\r\n at ExcelProcessScope \"Excel Process Scope\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/f0de1a4bc0.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/f0de1a4bc0.json deleted file mode 100644 index 0865f48397..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/f0de1a4bc0.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01", - "State": "Faulted", - "ProcessName": "PriceUpdater", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T09:12:40.420Z", - "EndTime": "2026-05-27T09:12:46.330Z", - "CreationTime": "2026-05-27T09:12:40.190Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/manifest.json deleted file mode 100644 index cb9dd650aa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Lookup Range price-list lookup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "f0de1a4bc0.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "f0de1a4bc0.json" - }, - { - "match": "or jobs get \"d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01\" --output json", - "file": "6bed206584.json" - }, - { - "match": "or jobs get d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01 --output json", - "file": "6bed206584.json" - }, - { - "match": "or jobs logs \"d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01\" --level Error --output json", - "file": "d3150e31e3.json" - }, - { - "match": "or jobs logs d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01 --level Error --output json", - "file": "d3150e31e3.json" - }, - { - "match": "or jobs logs \"d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01\" --output json", - "file": "c797d9ede2.json" - }, - { - "match": "or jobs logs d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01 --output json", - "file": "c797d9ede2.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/uip-fixture.json new file mode 100644 index 0000000000..8d8449cd78 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PriceUpdater\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T09:12:40.420Z\",\r\n \"EndTime\": \"2026-05-27T09:12:46.330Z\",\r\n \"CreationTime\": \"2026-05-27T09:12:40.190Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T09:12:46.262Z\",\r\n \"Message\": \"[Throw SKU Not Found] SKU-8842 not found in price list\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T09:12:46.305Z\",\r\n \"Message\": \"SKU-8842 not found in price list\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw SKU Not Found\\\"\\r\\n at If \\\"Guard empty result\\\"\\r\\n at Sequence \\\"Process Price List\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01\",\r\n \"StartTime\": \"2026-05-27T09:12:40.420Z\",\r\n \"EndTime\": \"2026-05-27T09:12:46.330Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"e7f8090a-1b2c-4d3e-4f5a-6b7c8d9e0f12\",\r\n \"Info\": \"Throw SKU Not Found: SKU-8842 not found in price list\\r\\n\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw SKU Not Found\\\"\\r\\n at If \\\"Guard empty result\\\"\\r\\n at Sequence \\\"Process Price List\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.BusinessRuleException: SKU-8842 not found in price list\",\r\n \"CreationTime\": \"2026-05-27T09:12:40.190Z\",\r\n \"ReleaseName\": \"PriceUpdater\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020188,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T09:12:46.330Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"BusinessRuleException\",\r\n \"TraceId\": \"d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01\",\r\n \"Id\": 682404519\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d6e7f809-0a1b-4c2d-3e4f-5a6b7c8d9e01", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T09:12:40.450Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T09:12:42.870Z\",\r\n \"Message\": \"[Use Excel File] Opened workbook data\\\\PriceList.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T09:12:44.110Z\",\r\n \"Message\": \"[Filter Status = Active] Applied filter on sheet 'Prices' column 'Status' = 'Active'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-27T09:12:45.880Z\",\r\n \"Message\": \"[Lookup Range] Searched sheet 'Prices' for 'SKU-8842'; Result = '' (no match in visible range)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T09:12:46.262Z\",\r\n \"Message\": \"[Throw SKU Not Found] SKU-8842 not found in price list\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T09:12:46.305Z\",\r\n \"Message\": \"SKU-8842 not found in price list\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw SKU Not Found\\\"\\r\\n at If \\\"Guard empty result\\\"\\r\\n at Sequence \\\"Process Price List\\\"\\r\\n at ExcelApplicationCard \\\"Use Excel File\\\"\\r\\n at ExcelProcessScope \\\"Excel Process Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T09:12:46.325Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/task.yaml index a4b541c7d3..a657ef8ee6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-active-filters/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/README.md new file mode 100644 index 0000000000..46e81c6881 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/README.md @@ -0,0 +1,55 @@ +# Lookup Range Failure - Workbook Locked / File In Use + +This scenario reproduces a classic `Lookup Range` failure caused by the +**workbook being held open by another process** on the robot host. The +surrounding `Excel Application Scope` cannot acquire the file handle and +faults at open with `System.IO.IOException: The process cannot access the +file '' because it is being used by another process` - before any +cell is read. + +## What this scenario uncovers + +**Root Cause:** The workbook is locked by another process on MOCK-HOST - +most often an **orphaned `EXCEL.EXE`** left behind by a prior unattended +run that did not dispose its Excel instance. The file looks closed to a +human but the handle is still held by a windowless process, so the next +run's `Excel Application Scope` cannot open it. The intermittent pattern +("some runs succeed, others fault") is the signature of handle contention, +not a workflow defect. + +This maps to: +`references/activity-packages/excel-activities/playbooks/workbook-file-locked.md` + +It is **distinct** from the modern-surface COM dispatcher fault +(`0x80010100 RPC_E_SYS_CALL_FAILED`): that one is a *blocked/hung* Excel +call; this one is *file-handle* contention. The error names the **file** +as in use, which routes here rather than to the COM-interop playbook. + +The user is framed as **off-host**, so the correct agent behavior is to +tie the IOException to a held file handle and hand over a host check list +(look for orphaned `EXCEL.EXE`, who/what holds the path) plus the fix - not +to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a classic `Excel Application Scope` + `Lookup Range` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `workbook-file-locked.md` (not the COM-interop playbook) +- Agent identified the workbook being held by another process (orphaned + `EXCEL.EXE` the most likely unattended cause) and recommended forcing a + clean release - e.g. a `Kill Process` (EXCEL) at workflow start plus a + cleanly-disposed scope, or serializing access - without fabricating host + actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/06ec89873f.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/06ec89873f.json deleted file mode 100644 index a465875e5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/06ec89873f.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f", - "State": "Faulted", - "ProcessName": "InvoiceLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T07:45:11.210Z", - "EndTime": "2026-05-27T07:45:14.530Z", - "CreationTime": "2026-05-27T07:45:10.980Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/5d8924f8cd.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/5d8924f8cd.json deleted file mode 100644 index ba6a8fd32b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/5d8924f8cd.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-27T07:45:11.240Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T07:45:12.690Z", - "Message": "[Excel Application Scope] Opening workbook C:\\ProgramData\\Automation\\Invoices.xlsx" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T07:45:14.402Z", - "Message": "[Excel Application Scope] The process cannot access the file 'C:\\ProgramData\\Automation\\Invoices.xlsx' because it is being used by another process." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T07:45:14.471Z", - "Message": "The process cannot access the file 'C:\\ProgramData\\Automation\\Invoices.xlsx' because it is being used by another process.\r\nin Main.xaml\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T07:45:14.515Z", - "Message": "Execution ended in: 00:00:03" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/6746227b6b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/6746227b6b.json deleted file mode 100644 index 6816594e30..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/6746227b6b.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-27T07:45:14.402Z", - "Message": "[Excel Application Scope] The process cannot access the file 'C:\\ProgramData\\Automation\\Invoices.xlsx' because it is being used by another process." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T07:45:14.471Z", - "Message": "The process cannot access the file 'C:\\ProgramData\\Automation\\Invoices.xlsx' because it is being used by another process.\r\nin Main.xaml\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/README.md deleted file mode 100644 index d038d747bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Lookup Range Failure - Workbook Locked / File In Use - -This scenario reproduces a classic `Lookup Range` failure caused by the -**workbook being held open by another process** on the robot host. The -surrounding `Excel Application Scope` cannot acquire the file handle and -faults at open with `System.IO.IOException: The process cannot access the -file '' because it is being used by another process` - before any -cell is read. - -## What this scenario uncovers - -**Root Cause:** The workbook is locked by another process on MOCK-HOST - -most often an **orphaned `EXCEL.EXE`** left behind by a prior unattended -run that did not dispose its Excel instance. The file looks closed to a -human but the handle is still held by a windowless process, so the next -run's `Excel Application Scope` cannot open it. The intermittent pattern -("some runs succeed, others fault") is the signature of handle contention, -not a workflow defect. - -This maps to: -`references/activity-packages/excel-activities/playbooks/workbook-file-locked.md` - -It is **distinct** from the modern-surface COM dispatcher fault -(`0x80010100 RPC_E_SYS_CALL_FAILED`): that one is a *blocked/hung* Excel -call; this one is *file-handle* contention. The error names the **file** -as in use, which routes here rather than to the COM-interop playbook. - -The user is framed as **off-host**, so the correct agent behavior is to -tie the IOException to a held file handle and hand over a host check list -(look for orphaned `EXCEL.EXE`, who/what holds the path) plus the fix - not -to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a classic `Excel Application Scope` + `Lookup Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `workbook-file-locked.md` (not the COM-interop playbook) -- Agent identified the workbook being held by another process (orphaned - `EXCEL.EXE` the most likely unattended cause) and recommended forcing a - clean release - e.g. a `Kill Process` (EXCEL) at workflow start plus a - cleanly-disposed scope, or serializing access - without fabricating host - actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/a79e514fa7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/a79e514fa7.json deleted file mode 100644 index 79fa49f013..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/a79e514fa7.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f", - "StartTime": "2026-05-27T07:45:11.210Z", - "EndTime": "2026-05-27T07:45:14.530Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f8091a2b-3c4d-4e5f-6071-8293a4b5c6d7", - "Info": "Excel Application Scope: The process cannot access the file 'C:\\ProgramData\\Automation\\Invoices.xlsx' because it is being used by another process.\r\n\r\nin Main.xaml\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.IOException: The process cannot access the file 'C:\\ProgramData\\Automation\\Invoices.xlsx' because it is being used by another process.", - "CreationTime": "2026-05-27T07:45:10.980Z", - "ReleaseName": "InvoiceLookup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020131, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T07:45:14.530Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f", - "Id": 682403914 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/manifest.json deleted file mode 100644 index 8e45505701..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Lookup Range invoice lookup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "06ec89873f.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "06ec89873f.json" - }, - { - "match": "or jobs get \"e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f\" --output json", - "file": "a79e514fa7.json" - }, - { - "match": "or jobs get e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f --output json", - "file": "a79e514fa7.json" - }, - { - "match": "or jobs logs \"e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f\" --level Error --output json", - "file": "6746227b6b.json" - }, - { - "match": "or jobs logs e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f --level Error --output json", - "file": "6746227b6b.json" - }, - { - "match": "or jobs logs \"e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f\" --output json", - "file": "5d8924f8cd.json" - }, - { - "match": "or jobs logs e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f --output json", - "file": "5d8924f8cd.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/uip-fixture.json new file mode 100644 index 0000000000..09e9444043 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T07:45:11.210Z\",\r\n \"EndTime\": \"2026-05-27T07:45:14.530Z\",\r\n \"CreationTime\": \"2026-05-27T07:45:10.980Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T07:45:14.402Z\",\r\n \"Message\": \"[Excel Application Scope] The process cannot access the file 'C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx' because it is being used by another process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T07:45:14.471Z\",\r\n \"Message\": \"The process cannot access the file 'C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx' because it is being used by another process.\\r\\nin Main.xaml\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f\",\r\n \"StartTime\": \"2026-05-27T07:45:11.210Z\",\r\n \"EndTime\": \"2026-05-27T07:45:14.530Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f8091a2b-3c4d-4e5f-6071-8293a4b5c6d7\",\r\n \"Info\": \"Excel Application Scope: The process cannot access the file 'C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx' because it is being used by another process.\\r\\n\\r\\nin Main.xaml\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.IOException: The process cannot access the file 'C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx' because it is being used by another process.\",\r\n \"CreationTime\": \"2026-05-27T07:45:10.980Z\",\r\n \"ReleaseName\": \"InvoiceLookup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020131,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T07:45:14.530Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f\",\r\n \"Id\": 682403914\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e7f8091a-2b3c-4d5e-9f60-7a8b9c0d1e2f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T07:45:11.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T07:45:12.690Z\",\r\n \"Message\": \"[Excel Application Scope] Opening workbook C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T07:45:14.402Z\",\r\n \"Message\": \"[Excel Application Scope] The process cannot access the file 'C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx' because it is being used by another process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T07:45:14.471Z\",\r\n \"Message\": \"The process cannot access the file 'C:\\\\ProgramData\\\\Automation\\\\Invoices.xlsx' because it is being used by another process.\\r\\nin Main.xaml\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T07:45:14.515Z\",\r\n \"Message\": \"Execution ended in: 00:00:03\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/task.yaml index 3110594420..83ddc3c448 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-file-locked/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/README.md new file mode 100644 index 0000000000..af185f0b4a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/README.md @@ -0,0 +1,54 @@ +# Lookup Range Failure - Silent Miss Against a Formula Cell + +This scenario reproduces a `Lookup Range` "value not found" failure where the +searched value is physically present in the sheet but the target cell is the +**computed result of an Excel formula** (a cross-sheet VLOOKUP). The classic +Interop read returns the cell's cached calculated value; when the cache is +stale or empty under the unattended robot's headless Excel session, the +lookup returns empty even though the cell visibly displays the SKU when +opened by hand. + +## What this scenario uncovers + +**Root Cause:** `Main.xaml` runs a classic `Lookup Range` +(`UiPath.Excel.Activities.ExcelLookUpRange`) inside an `Excel Application +Scope` on the `PublishedPrices` sheet of `DailyPrices.xlsx`. There is **no +filter** in the workflow. The `PublishedPrices.SKU` column is built by a +formula -- a VLOOKUP against an upstream `SourceFeed` sheet refreshed by a +separate job. Excel's Interop API returns the cell's *cached* calculated +value; for a freshly-recomputed cross-sheet VLOOKUP the cache can be stale +or empty in the robot's headless session, so the Interop read returns no +value and `Lookup Range` cannot match `SKU-7392`. The workflow's guard then +throws "not found". The value is genuinely present in the sheet -- the +Interop read of the formula's cached value just does not see it. + +This maps to: +`references/activity-packages/excel-activities/playbooks/lookup-range-formula-cells.md` + +The cause is **partly discoverable in Main.xaml** (absence of any filter +rules out the active-filters playbook) and **partly via the user** (the +persona confirms the SKU column is a VLOOKUP when the agent asks how the +column is populated). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/Main.xaml` | Excel Application Scope -> Lookup Range (SKU-7392) -> Throw-if-empty (no filter) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `lookup-range-formula-cells.md` +- Agent identified the formula/cache cause (cell's calculated value, Interop + reads cached value that is stale/empty) and recommended either converting + formulas to static values (Paste Special > Values) OR migrating to the + Workbook `Read Range` + `Lookup Data Table` (OpenXML) path + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1070f5ff6c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1070f5ff6c.json deleted file mode 100644 index 8ecb8408bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1070f5ff6c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-27T14:38:17.534Z", - "Message": "[Throw SKU Not Found] Today's published price for SKU-7392 not found in PublishedPrices sheet" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T14:38:17.582Z", - "Message": "Today's published price for SKU-7392 not found in PublishedPrices sheet\r\nin Main.xaml\r\n at Throw \"Throw SKU Not Found\"\r\n at If \"Guard empty result\"\r\n at Sequence \"Process Published Prices\"\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/17089fd2d2.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/17089fd2d2.json deleted file mode 100644 index bf377d6637..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/17089fd2d2.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-27T14:38:12.220Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T14:38:14.060Z", - "Message": "[Excel Application Scope] Opened workbook data\\DailyPrices.xlsx" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-27T14:38:16.940Z", - "Message": "[Lookup Range] Searched sheet 'PublishedPrices' for 'SKU-7392'; Result = '' (no match)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T14:38:17.534Z", - "Message": "[Throw SKU Not Found] Today's published price for SKU-7392 not found in PublishedPrices sheet" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T14:38:17.582Z", - "Message": "Today's published price for SKU-7392 not found in PublishedPrices sheet\r\nin Main.xaml\r\n at Throw \"Throw SKU Not Found\"\r\n at If \"Guard empty result\"\r\n at Sequence \"Process Published Prices\"\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T14:38:17.602Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1a8f047d1c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1a8f047d1c.json deleted file mode 100644 index aebc8abb71..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1a8f047d1c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c", - "State": "Faulted", - "ProcessName": "DailyPriceSync", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T14:38:12.180Z", - "EndTime": "2026-05-27T14:38:17.610Z", - "CreationTime": "2026-05-27T14:38:11.940Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1d5e44fd8d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1d5e44fd8d.json deleted file mode 100644 index ec2eae68e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/1d5e44fd8d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c", - "StartTime": "2026-05-27T14:38:12.180Z", - "EndTime": "2026-05-27T14:38:17.610Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "e3f4a5b6-7c8d-49e1-8f02-3a4b5c6d7e8f", - "Info": "Throw SKU Not Found: Today's published price for SKU-7392 not found in PublishedPrices sheet\r\n\r\nin Main.xaml\r\n at Throw \"Throw SKU Not Found\"\r\n at If \"Guard empty result\"\r\n at Sequence \"Process Published Prices\"\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.BusinessRuleException: Today's published price for SKU-7392 not found in PublishedPrices sheet", - "CreationTime": "2026-05-27T14:38:11.940Z", - "ReleaseName": "DailyPriceSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020412, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "schedule", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T14:38:17.610Z", - "JobError": {}, - "ErrorCode": "BusinessRuleException", - "TraceId": "f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c", - "Id": 682405188 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/README.md deleted file mode 100644 index eca25398a8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Lookup Range Failure - Silent Miss Against a Formula Cell - -This scenario reproduces a `Lookup Range` "value not found" failure where the -searched value is physically present in the sheet but the target cell is the -**computed result of an Excel formula** (a cross-sheet VLOOKUP). The classic -Interop read returns the cell's cached calculated value; when the cache is -stale or empty under the unattended robot's headless Excel session, the -lookup returns empty even though the cell visibly displays the SKU when -opened by hand. - -## What this scenario uncovers - -**Root Cause:** `Main.xaml` runs a classic `Lookup Range` -(`UiPath.Excel.Activities.ExcelLookUpRange`) inside an `Excel Application -Scope` on the `PublishedPrices` sheet of `DailyPrices.xlsx`. There is **no -filter** in the workflow. The `PublishedPrices.SKU` column is built by a -formula -- a VLOOKUP against an upstream `SourceFeed` sheet refreshed by a -separate job. Excel's Interop API returns the cell's *cached* calculated -value; for a freshly-recomputed cross-sheet VLOOKUP the cache can be stale -or empty in the robot's headless session, so the Interop read returns no -value and `Lookup Range` cannot match `SKU-7392`. The workflow's guard then -throws "not found". The value is genuinely present in the sheet -- the -Interop read of the formula's cached value just does not see it. - -This maps to: -`references/activity-packages/excel-activities/playbooks/lookup-range-formula-cells.md` - -The cause is **partly discoverable in Main.xaml** (absence of any filter -rules out the active-filters playbook) and **partly via the user** (the -persona confirms the SKU column is a VLOOKUP when the agent asks how the -column is populated). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/Main.xaml` | Excel Application Scope -> Lookup Range (SKU-7392) -> Throw-if-empty (no filter) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `lookup-range-formula-cells.md` -- Agent identified the formula/cache cause (cell's calculated value, Interop - reads cached value that is stale/empty) and recommended either converting - formulas to static values (Paste Special > Values) OR migrating to the - Workbook `Read Range` + `Lookup Data Table` (OpenXML) path diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/manifest.json deleted file mode 100644 index 3fe4834d35..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Lookup Range daily price-sync scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "1a8f047d1c.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "1a8f047d1c.json" - }, - { - "match": "or jobs get \"f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c\" --output json", - "file": "1d5e44fd8d.json" - }, - { - "match": "or jobs get f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c --output json", - "file": "1d5e44fd8d.json" - }, - { - "match": "or jobs logs \"f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c\" --level Error --output json", - "file": "1070f5ff6c.json" - }, - { - "match": "or jobs logs f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c --level Error --output json", - "file": "1070f5ff6c.json" - }, - { - "match": "or jobs logs \"f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c\" --output json", - "file": "17089fd2d2.json" - }, - { - "match": "or jobs logs f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c --output json", - "file": "17089fd2d2.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/uip-fixture.json new file mode 100644 index 0000000000..4b6303f630 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyPriceSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T14:38:12.180Z\",\r\n \"EndTime\": \"2026-05-27T14:38:17.610Z\",\r\n \"CreationTime\": \"2026-05-27T14:38:11.940Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T14:38:17.534Z\",\r\n \"Message\": \"[Throw SKU Not Found] Today's published price for SKU-7392 not found in PublishedPrices sheet\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T14:38:17.582Z\",\r\n \"Message\": \"Today's published price for SKU-7392 not found in PublishedPrices sheet\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw SKU Not Found\\\"\\r\\n at If \\\"Guard empty result\\\"\\r\\n at Sequence \\\"Process Published Prices\\\"\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c\",\r\n \"StartTime\": \"2026-05-27T14:38:12.180Z\",\r\n \"EndTime\": \"2026-05-27T14:38:17.610Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"e3f4a5b6-7c8d-49e1-8f02-3a4b5c6d7e8f\",\r\n \"Info\": \"Throw SKU Not Found: Today's published price for SKU-7392 not found in PublishedPrices sheet\\r\\n\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw SKU Not Found\\\"\\r\\n at If \\\"Guard empty result\\\"\\r\\n at Sequence \\\"Process Published Prices\\\"\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.BusinessRuleException: Today's published price for SKU-7392 not found in PublishedPrices sheet\",\r\n \"CreationTime\": \"2026-05-27T14:38:11.940Z\",\r\n \"ReleaseName\": \"DailyPriceSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020412,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"schedule\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T14:38:17.610Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"BusinessRuleException\",\r\n \"TraceId\": \"f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c\",\r\n \"Id\": 682405188\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f1a2b3c4-5d6e-4789-9b0c-1d2e3f4a5b6c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T14:38:12.220Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T14:38:14.060Z\",\r\n \"Message\": \"[Excel Application Scope] Opened workbook data\\\\DailyPrices.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-27T14:38:16.940Z\",\r\n \"Message\": \"[Lookup Range] Searched sheet 'PublishedPrices' for 'SKU-7392'; Result = '' (no match)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T14:38:17.534Z\",\r\n \"Message\": \"[Throw SKU Not Found] Today's published price for SKU-7392 not found in PublishedPrices sheet\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T14:38:17.582Z\",\r\n \"Message\": \"Today's published price for SKU-7392 not found in PublishedPrices sheet\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw SKU Not Found\\\"\\r\\n at If \\\"Guard empty result\\\"\\r\\n at Sequence \\\"Process Published Prices\\\"\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T14:38:17.602Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/task.yaml index 0dfbf22ed8..b076fa3846 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-formula-cells/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/README.md new file mode 100644 index 0000000000..6de6895392 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/README.md @@ -0,0 +1,49 @@ +# Lookup Range Failure - Invalid Range (Empty-String Literal) + +This scenario reproduces a classic `Lookup Range` failure caused by the +**`Range` property being set to a literal empty string `""`** instead of +being left blank. For `Lookup Range`, a whole-sheet search is expressed by +leaving the `Range` field **completely empty** (no value at all); a literal +`""` is an *invalid range value* and faults the activity with +`The range '' is not valid` once the workbook is open. + +## What this scenario uncovers + +**Root Cause:** In `Main.xaml`, the `Lookup Range` activity has +`Range=""""` - the C# expression `""`, which evaluates to an +empty string. The `Excel Application Scope` opens `Catalog.xlsx` +successfully, then the `Lookup Range` faults at the range parse step +because `""` is not a valid range. The whole-sheet search the workflow +intends requires a **blank** `Range` field, not the empty-string literal. + +This maps to: +`references/activity-packages/excel-activities/playbooks/lookup-range-invalid-range.md` + +The empty-string-vs-empty-field distinction is the single most common form +of this failure, and it is discoverable in `Main.xaml`: the scope opens +fine (the log shows "Workbook opened"), then the `Lookup Range` errors with +`The range '' is not valid`, and the activity's `Range` is set to the +literal `""`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a classic `Excel Application Scope` + `Lookup Range` whose `Range` is set to `""` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `lookup-range-invalid-range.md` +- Agent identified the `Range` set to a literal empty string `""` (rather + than left blank) as the cause, found in `Main.xaml`, and recommended + clearing the `Range` field entirely for a whole-sheet search (or setting + a valid A1 reference) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/117a546c1b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/117a546c1b.json deleted file mode 100644 index 8fdadf9455..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/117a546c1b.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:02:37.741Z", - "Message": "[Lookup Range] The range '' is not valid." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:02:37.808Z", - "Message": "Lookup Range: The range '' is not valid.\r\nin Main.xaml\r\n at ExcelLookUpRange \"Lookup Range\"\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/38e0209800.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/38e0209800.json deleted file mode 100644 index 87ac027f20..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/38e0209800.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:02:33.440Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:02:35.020Z", - "Message": "[Excel Application Scope] Opening workbook C:\\ProgramData\\Automation\\Catalog.xlsx" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:02:36.380Z", - "Message": "[Excel Application Scope] Workbook opened" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:02:37.741Z", - "Message": "[Lookup Range] The range '' is not valid." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:02:37.808Z", - "Message": "Lookup Range: The range '' is not valid.\r\nin Main.xaml\r\n at ExcelLookUpRange \"Lookup Range\"\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:02:37.855Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/README.md deleted file mode 100644 index c50241f181..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Lookup Range Failure - Invalid Range (Empty-String Literal) - -This scenario reproduces a classic `Lookup Range` failure caused by the -**`Range` property being set to a literal empty string `""`** instead of -being left blank. For `Lookup Range`, a whole-sheet search is expressed by -leaving the `Range` field **completely empty** (no value at all); a literal -`""` is an *invalid range value* and faults the activity with -`The range '' is not valid` once the workbook is open. - -## What this scenario uncovers - -**Root Cause:** In `Main.xaml`, the `Lookup Range` activity has -`Range=""""` - the C# expression `""`, which evaluates to an -empty string. The `Excel Application Scope` opens `Catalog.xlsx` -successfully, then the `Lookup Range` faults at the range parse step -because `""` is not a valid range. The whole-sheet search the workflow -intends requires a **blank** `Range` field, not the empty-string literal. - -This maps to: -`references/activity-packages/excel-activities/playbooks/lookup-range-invalid-range.md` - -The empty-string-vs-empty-field distinction is the single most common form -of this failure, and it is discoverable in `Main.xaml`: the scope opens -fine (the log shows "Workbook opened"), then the `Lookup Range` errors with -`The range '' is not valid`, and the activity's `Range` is set to the -literal `""`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a classic `Excel Application Scope` + `Lookup Range` whose `Range` is set to `""` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `lookup-range-invalid-range.md` -- Agent identified the `Range` set to a literal empty string `""` (rather - than left blank) as the cause, found in `Main.xaml`, and recommended - clearing the `Range` field entirely for a whole-sheet search (or setting - a valid A1 reference) diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/bd39fa1444.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/bd39fa1444.json deleted file mode 100644 index b36cc8b36d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/bd39fa1444.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8", - "State": "Faulted", - "ProcessName": "CatalogLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T08:02:33.410Z", - "EndTime": "2026-05-27T08:02:37.870Z", - "CreationTime": "2026-05-27T08:02:33.180Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/c62720870a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/c62720870a.json deleted file mode 100644 index 641f7aaff3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/c62720870a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8", - "StartTime": "2026-05-27T08:02:33.410Z", - "EndTime": "2026-05-27T08:02:37.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b2c3d4e5-6f70-4819-8293-a4b5c6d7e8f9", - "Info": "Lookup Range: The range '' is not valid.\r\n\r\nin Main.xaml\r\n at ExcelLookUpRange \"Lookup Range\"\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Excel.ExcelException: The range '' is not valid.", - "CreationTime": "2026-05-27T08:02:33.180Z", - "ReleaseName": "CatalogLookup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020142, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T08:02:37.870Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8", - "Id": 682405288 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/manifest.json deleted file mode 100644 index 979aa89438..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Lookup Range catalog lookup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "bd39fa1444.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "bd39fa1444.json" - }, - { - "match": "or jobs get \"a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8\" --output json", - "file": "c62720870a.json" - }, - { - "match": "or jobs get a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8 --output json", - "file": "c62720870a.json" - }, - { - "match": "or jobs logs \"a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8\" --level Error --output json", - "file": "117a546c1b.json" - }, - { - "match": "or jobs logs a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8 --level Error --output json", - "file": "117a546c1b.json" - }, - { - "match": "or jobs logs \"a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8\" --output json", - "file": "38e0209800.json" - }, - { - "match": "or jobs logs a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8 --output json", - "file": "38e0209800.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/uip-fixture.json new file mode 100644 index 0000000000..f114fd4356 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CatalogLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T08:02:33.410Z\",\r\n \"EndTime\": \"2026-05-27T08:02:37.870Z\",\r\n \"CreationTime\": \"2026-05-27T08:02:33.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:02:37.741Z\",\r\n \"Message\": \"[Lookup Range] The range '' is not valid.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:02:37.808Z\",\r\n \"Message\": \"Lookup Range: The range '' is not valid.\\r\\nin Main.xaml\\r\\n at ExcelLookUpRange \\\"Lookup Range\\\"\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8\",\r\n \"StartTime\": \"2026-05-27T08:02:33.410Z\",\r\n \"EndTime\": \"2026-05-27T08:02:37.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b2c3d4e5-6f70-4819-8293-a4b5c6d7e8f9\",\r\n \"Info\": \"Lookup Range: The range '' is not valid.\\r\\n\\r\\nin Main.xaml\\r\\n at ExcelLookUpRange \\\"Lookup Range\\\"\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Excel.ExcelException: The range '' is not valid.\",\r\n \"CreationTime\": \"2026-05-27T08:02:33.180Z\",\r\n \"ReleaseName\": \"CatalogLookup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020142,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T08:02:37.870Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8\",\r\n \"Id\": 682405288\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-5e6f-4708-8192-a3b4c5d6e7f8", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:02:33.440Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:02:35.020Z\",\r\n \"Message\": \"[Excel Application Scope] Opening workbook C:\\\\ProgramData\\\\Automation\\\\Catalog.xlsx\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:02:36.380Z\",\r\n \"Message\": \"[Excel Application Scope] Workbook opened\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:02:37.741Z\",\r\n \"Message\": \"[Lookup Range] The range '' is not valid.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:02:37.808Z\",\r\n \"Message\": \"Lookup Range: The range '' is not valid.\\r\\nin Main.xaml\\r\\n at ExcelLookUpRange \\\"Lookup Range\\\"\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:02:37.855Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/task.yaml index b8e2a84546..c43dcfdf28 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-invalid-range/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/README.md new file mode 100644 index 0000000000..7f05de43bf --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/README.md @@ -0,0 +1,47 @@ +# Lookup Range Failure - Excel Not Installed on the Robot Host + +This scenario reproduces a classic `Lookup Range` failure caused by +**Microsoft Excel not being installed** on the execution machine. The +classic activity drives the Excel Interop API, so the surrounding +`Excel Application Scope` cannot create the `Excel.Application` COM +object and faults at startup with `REGDB_E_CLASSNOTREG` (0x80040154). + +## What this scenario uncovers + +**Root Cause:** The classic `Lookup Range` (`UiPath.Excel.Activities.ExcelLookUpRange`) +inside an `Excel Application Scope` needs a registered desktop Excel +install. The new unattended robot host has no Excel, so the COM class +factory for `Excel.Application` cannot be created and the scope faults +before any cell is read. The "worked on dev, broke after the move to the +new robot" detail points at a host-environment cause, not a workflow +defect. + +This maps to: +`references/activity-packages/excel-activities/playbooks/excel-not-installed.md` + +The user is framed as **off-host**, so the correct agent behavior is to +tie the HRESULT to a missing Excel install and recommend either +installing Excel or migrating to the Excel-free Workbook path - not to +attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a classic `Excel Application Scope` + `Lookup Range` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `excel-not-installed.md` and used its Lookup Range discriminator +- Agent identified "Excel not installed on the robot host" as the cause + and recommended either installing Excel or migrating to Workbook + Read Range + Lookup Data Table (either fix path scores full marks) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/275773fd83.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/275773fd83.json deleted file mode 100644 index 6ee15f50e3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/275773fd83.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:30:04.120Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:30:05.640Z", - "Message": "[Excel Application Scope] Starting Microsoft Excel" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:30:06.071Z", - "Message": "[Excel Application Scope] Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG))." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:30:06.118Z", - "Message": "Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\nin Main.xaml\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:30:06.135Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/4e8b1f73bc.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/4e8b1f73bc.json deleted file mode 100644 index e7ba4a9b81..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/4e8b1f73bc.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:30:06.071Z", - "Message": "[Excel Application Scope] Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG))." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:30:06.118Z", - "Message": "Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\nin Main.xaml\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/5019238b56.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/5019238b56.json deleted file mode 100644 index 14a2949d82..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/5019238b56.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80", - "State": "Faulted", - "ProcessName": "VendorLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T08:30:04.090Z", - "EndTime": "2026-05-27T08:30:06.140Z", - "CreationTime": "2026-05-27T08:30:03.860Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/73f7c45f4b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/73f7c45f4b.json deleted file mode 100644 index 6b75cca4eb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/73f7c45f4b.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80", - "StartTime": "2026-05-27T08:30:04.090Z", - "EndTime": "2026-05-27T08:30:06.140Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d5e6f708-9a0b-4c1d-2e3f-4a5b6c7d8e90", - "Info": "Excel Application Scope: Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\n\r\nin Main.xaml\r\n at ExcelApplicationScope \"Excel Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).", - "CreationTime": "2026-05-27T08:30:03.860Z", - "ReleaseName": "VendorLookup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020114, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T08:30:06.140Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80", - "Id": 682401773 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/README.md deleted file mode 100644 index c4eb871c6a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Lookup Range Failure - Excel Not Installed on the Robot Host - -This scenario reproduces a classic `Lookup Range` failure caused by -**Microsoft Excel not being installed** on the execution machine. The -classic activity drives the Excel Interop API, so the surrounding -`Excel Application Scope` cannot create the `Excel.Application` COM -object and faults at startup with `REGDB_E_CLASSNOTREG` (0x80040154). - -## What this scenario uncovers - -**Root Cause:** The classic `Lookup Range` (`UiPath.Excel.Activities.ExcelLookUpRange`) -inside an `Excel Application Scope` needs a registered desktop Excel -install. The new unattended robot host has no Excel, so the COM class -factory for `Excel.Application` cannot be created and the scope faults -before any cell is read. The "worked on dev, broke after the move to the -new robot" detail points at a host-environment cause, not a workflow -defect. - -This maps to: -`references/activity-packages/excel-activities/playbooks/excel-not-installed.md` - -The user is framed as **off-host**, so the correct agent behavior is to -tie the HRESULT to a missing Excel install and recommend either -installing Excel or migrating to the Excel-free Workbook path - not to -attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a classic `Excel Application Scope` + `Lookup Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `excel-not-installed.md` and used its Lookup Range discriminator -- Agent identified "Excel not installed on the robot host" as the cause - and recommended either installing Excel or migrating to Workbook - Read Range + Lookup Data Table (either fix path scores full marks) diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/manifest.json deleted file mode 100644 index 521d28e9b4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Lookup Range vendor lookup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "5019238b56.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "5019238b56.json" - }, - { - "match": "or jobs get \"c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80\" --output json", - "file": "73f7c45f4b.json" - }, - { - "match": "or jobs get c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80 --output json", - "file": "73f7c45f4b.json" - }, - { - "match": "or jobs logs \"c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80\" --level Error --output json", - "file": "4e8b1f73bc.json" - }, - { - "match": "or jobs logs c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80 --level Error --output json", - "file": "4e8b1f73bc.json" - }, - { - "match": "or jobs logs \"c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80\" --output json", - "file": "275773fd83.json" - }, - { - "match": "or jobs logs c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80 --output json", - "file": "275773fd83.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/uip-fixture.json new file mode 100644 index 0000000000..a9ecd55ba7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VendorLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T08:30:04.090Z\",\r\n \"EndTime\": \"2026-05-27T08:30:06.140Z\",\r\n \"CreationTime\": \"2026-05-27T08:30:03.860Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:30:06.071Z\",\r\n \"Message\": \"[Excel Application Scope] Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:30:06.118Z\",\r\n \"Message\": \"Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\nin Main.xaml\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80\",\r\n \"StartTime\": \"2026-05-27T08:30:04.090Z\",\r\n \"EndTime\": \"2026-05-27T08:30:06.140Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d5e6f708-9a0b-4c1d-2e3f-4a5b6c7d8e90\",\r\n \"Info\": \"Excel Application Scope: Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\n\\r\\nin Main.xaml\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\",\r\n \"CreationTime\": \"2026-05-27T08:30:03.860Z\",\r\n \"ReleaseName\": \"VendorLookup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020114,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T08:30:06.140Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80\",\r\n \"Id\": 682401773\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c4d5e6f7-8a9b-4c0d-1e2f-3a4b5c6d7e80", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:30:04.120Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:30:05.640Z\",\r\n \"Message\": \"[Excel Application Scope] Starting Microsoft Excel\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:30:06.071Z\",\r\n \"Message\": \"[Excel Application Scope] Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:30:06.118Z\",\r\n \"Message\": \"Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\nin Main.xaml\\r\\n at ExcelApplicationScope \\\"Excel Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:30:06.135Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/task.yaml index 41d8cf6ae8..74ab99704d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-not-installed/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/README.md new file mode 100644 index 0000000000..0127cf1380 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/README.md @@ -0,0 +1,52 @@ +# Lookup Range Failure - Object Reference Not Set (No Excel Scope) + +This scenario reproduces a classic `Lookup Range` failure caused by the +activity being placed **outside any Excel scope**. The classic +`Lookup Range` needs an `Excel Application Scope` to provide its workbook +context; dropped directly into the Main `Sequence` with no scope, it +dereferences a null workbook handle and faults with +`System.NullReferenceException: Object reference not set to an instance of +an object`. + +## What this scenario uncovers + +**Root Cause:** In `Main.xaml`, the `Lookup Range` activity sits directly +in the Main `Sequence` with **no surrounding `Excel Application Scope`** +(or `Use Excel File`). There is no workbook context object, so the activity +dereferences null and throws a `NullReferenceException` the instant it runs. + +This maps to: +`references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` +(the **no surrounding scope** cause). + +It is **not** a missing Excel install (no `REGDB_E_CLASSNOTREG` -> that +routes to `excel-not-installed.md`) and **not** a file-in-use +fault (no IOException -> `workbook-file-locked.md`). The fault stack +goes `ExcelLookUpRange` -> `Sequence "Main Sequence"` -> `Main` with **no scope +in between**, which is the tell that the activity has no workbook context. + +The misconfiguration is discoverable in `Main.xaml`: the `ExcelLookUpRange` is +not wrapped in any Excel scope. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Lookup Range` placed outside any Excel scope | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `null-reference-from-activity.md` +- Agent identified that the `Lookup Range` is outside any Excel scope (no + workbook context), found in `Main.xaml`, as the cause of the + `NullReferenceException`, and recommended wrapping it in an + `Excel Application Scope` (or `Use Excel File`) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/0298986d61.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/0298986d61.json deleted file mode 100644 index ddd8704db2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/0298986d61.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f7081929-3a4b-4c5d-8e6f-708192a3b4c5", - "StartTime": "2026-05-27T08:18:55.640Z", - "EndTime": "2026-05-27T08:18:57.900Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "081929a3-4b5c-4d6e-8f70-8192a3b4c5d6", - "Info": "Lookup Range: Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at ExcelLookUpRange \"Lookup Range\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.", - "CreationTime": "2026-05-27T08:18:55.410Z", - "ReleaseName": "OrderLookup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020156, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-27T08:18:57.900Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f7081929-3a4b-4c5d-8e6f-708192a3b4c5", - "Id": 682406701 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/1135c96083.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/1135c96083.json deleted file mode 100644 index 2e0b14bb47..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/1135c96083.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f7081929-3a4b-4c5d-8e6f-708192a3b4c5", - "State": "Faulted", - "ProcessName": "OrderLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T08:18:55.640Z", - "EndTime": "2026-05-27T08:18:57.900Z", - "CreationTime": "2026-05-27T08:18:55.410Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/38efe4449c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/38efe4449c.json deleted file mode 100644 index 10a9063a1e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/38efe4449c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:18:55.670Z", - "Message": "Execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:18:57.771Z", - "Message": "[Lookup Range] Object reference not set to an instance of an object." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:18:57.838Z", - "Message": "Lookup Range: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at ExcelLookUpRange \"Lookup Range\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-27T08:18:57.884Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/6e7db758ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/6e7db758ab.json deleted file mode 100644 index 7c6174039e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/6e7db758ab.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/README.md deleted file mode 100644 index eb89da1ee8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Lookup Range Failure - Object Reference Not Set (No Excel Scope) - -This scenario reproduces a classic `Lookup Range` failure caused by the -activity being placed **outside any Excel scope**. The classic -`Lookup Range` needs an `Excel Application Scope` to provide its workbook -context; dropped directly into the Main `Sequence` with no scope, it -dereferences a null workbook handle and faults with -`System.NullReferenceException: Object reference not set to an instance of -an object`. - -## What this scenario uncovers - -**Root Cause:** In `Main.xaml`, the `Lookup Range` activity sits directly -in the Main `Sequence` with **no surrounding `Excel Application Scope`** -(or `Use Excel File`). There is no workbook context object, so the activity -dereferences null and throws a `NullReferenceException` the instant it runs. - -This maps to: -`references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` -(the **no surrounding scope** cause). - -It is **not** a missing Excel install (no `REGDB_E_CLASSNOTREG` -> that -routes to `excel-not-installed.md`) and **not** a file-in-use -fault (no IOException -> `workbook-file-locked.md`). The fault stack -goes `ExcelLookUpRange` -> `Sequence "Main Sequence"` -> `Main` with **no scope -in between**, which is the tell that the activity has no workbook context. - -The misconfiguration is discoverable in `Main.xaml`: the `ExcelLookUpRange` is -not wrapped in any Excel scope. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Lookup Range` placed outside any Excel scope | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `null-reference-from-activity.md` -- Agent identified that the `Lookup Range` is outside any Excel scope (no - workbook context), found in `Main.xaml`, as the cause of the - `NullReferenceException`, and recommended wrapping it in an - `Excel Application Scope` (or `Use Excel File`) diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/dda676fc8f.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/dda676fc8f.json deleted file mode 100644 index 27a08412d2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/dda676fc8f.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:18:57.771Z", - "Message": "[Lookup Range] Object reference not set to an instance of an object." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-27T08:18:57.838Z", - "Message": "Lookup Range: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at ExcelLookUpRange \"Lookup Range\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/manifest.json deleted file mode 100644 index fea6b03d37..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Lookup Range order lookup scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6e7db758ab.json" - }, - { - "match": "or jobs list --folder-key \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\" --state Faulted --output json", - "file": "1135c96083.json" - }, - { - "match": "or jobs list --folder-key b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e --state Faulted --output json", - "file": "1135c96083.json" - }, - { - "match": "or jobs get \"f7081929-3a4b-4c5d-8e6f-708192a3b4c5\" --output json", - "file": "0298986d61.json" - }, - { - "match": "or jobs get f7081929-3a4b-4c5d-8e6f-708192a3b4c5 --output json", - "file": "0298986d61.json" - }, - { - "match": "or jobs logs \"f7081929-3a4b-4c5d-8e6f-708192a3b4c5\" --level Error --output json", - "file": "dda676fc8f.json" - }, - { - "match": "or jobs logs f7081929-3a4b-4c5d-8e6f-708192a3b4c5 --level Error --output json", - "file": "dda676fc8f.json" - }, - { - "match": "or jobs logs \"f7081929-3a4b-4c5d-8e6f-708192a3b4c5\" --output json", - "file": "38efe4449c.json" - }, - { - "match": "or jobs logs f7081929-3a4b-4c5d-8e6f-708192a3b4c5 --output json", - "file": "38efe4449c.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/uip-fixture.json new file mode 100644 index 0000000000..5546586200 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7081929-3a4b-4c5d-8e6f-708192a3b4c5\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T08:18:55.640Z\",\r\n \"EndTime\": \"2026-05-27T08:18:57.900Z\",\r\n \"CreationTime\": \"2026-05-27T08:18:55.410Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7081929-3a4b-4c5d-8e6f-708192a3b4c5", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:18:57.771Z\",\r\n \"Message\": \"[Lookup Range] Object reference not set to an instance of an object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:18:57.838Z\",\r\n \"Message\": \"Lookup Range: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ExcelLookUpRange \\\"Lookup Range\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f7081929-3a4b-4c5d-8e6f-708192a3b4c5", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f7081929-3a4b-4c5d-8e6f-708192a3b4c5\",\r\n \"StartTime\": \"2026-05-27T08:18:55.640Z\",\r\n \"EndTime\": \"2026-05-27T08:18:57.900Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"081929a3-4b5c-4d6e-8f70-8192a3b4c5d6\",\r\n \"Info\": \"Lookup Range: Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at ExcelLookUpRange \\\"Lookup Range\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"CreationTime\": \"2026-05-27T08:18:55.410Z\",\r\n \"ReleaseName\": \"OrderLookup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020156,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-27T08:18:57.900Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f7081929-3a4b-4c5d-8e6f-708192a3b4c5\",\r\n \"Id\": 682406701\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7081929-3a4b-4c5d-8e6f-708192a3b4c5", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:18:55.670Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:18:57.771Z\",\r\n \"Message\": \"[Lookup Range] Object reference not set to an instance of an object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-27T08:18:57.838Z\",\r\n \"Message\": \"Lookup Range: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ExcelLookUpRange \\\"Lookup Range\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-27T08:18:57.884Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c9d4e7-3a8f-4b1d-9e5c-7f0a2b3c4d5e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/task.yaml index ecd016c784..c95ad4b2b3 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-lookuprange-null-reference/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/README.md new file mode 100644 index 0000000000..e31315416f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/README.md @@ -0,0 +1,45 @@ +# Excel Execute Macro — Macro Name Not Found + +This scenario reproduces an Execute Macro failure where the configured +`MacroName` does not exist in the workbook's VBA project. The job +ends with: + +``` +System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's Execute Macro activity is configured +with `MacroName: "RunImport"`. The workbook's VBA project contains +macros `Module1.ProcessData` and `Module2.UpdateReport` — NOT +`RunImport`. The error string is the canonical Excel-COM "macro not +in workbook OR macros disabled" message; the distinguishing evidence +is the activity's VBA-module enumeration in the logs, which shows the +absence of `RunImport`. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/execute-macro-failures.md` +(the "Macro name not found in the workbook" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelMacroProcess` project — `Use Excel File` → `Execute Macro` with `MacroName: "RunImport"` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (canonical "Cannot run the +macro" COMException) → `jobs logs` (VBA-module enumeration is the +smoking gun) → workflow source review (confirms configured +`MacroName: "RunImport"`) → conclude branch 1. + +> **Note on fixtures.** Synthetic. The macro names and module +> names are placeholders. The test grades whether the agent +> distinguishes branch 1 (macro absent) from branch 5 (macro +> present but Trust Center disabled) — the absence-from-enumeration +> evidence is decisive. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/3412bc7963.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/3412bc7963.json deleted file mode 100644 index 2e9d775598..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/3412bc7963.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa1111cc-001", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-002", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelMacroProcess] Starting daily import — running macro RunImport on C:\\Robot\\Data\\sales-macro.xlsm", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-003", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-macro.xlsm — workbook opened", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-004", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.110Z", - "Message": "Execute Macro: RunImport — enumerating VBA modules in workbook", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-005", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.310Z", - "Message": "Execute Macro: RunImport — VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-006", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.510Z", - "Message": "Execute Macro: RunImport — dispatching macro via COM (Application.Run 'RunImport')", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111cc-007", - "JobKey": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Execute Macro: RunImport — System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled.", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/README.md deleted file mode 100644 index fca7b5347f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Excel Execute Macro — Macro Name Not Found - -This scenario reproduces an Execute Macro failure where the configured -`MacroName` does not exist in the workbook's VBA project. The job -ends with: - -``` -System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's Execute Macro activity is configured -with `MacroName: "RunImport"`. The workbook's VBA project contains -macros `Module1.ProcessData` and `Module2.UpdateReport` — NOT -`RunImport`. The error string is the canonical Excel-COM "macro not -in workbook OR macros disabled" message; the distinguishing evidence -is the activity's VBA-module enumeration in the logs, which shows the -absence of `RunImport`. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/execute-macro-failures.md` -(the "Macro name not found in the workbook" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelMacroProcess` project — `Use Excel File` → `Execute Macro` with `MacroName: "RunImport"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entry from `Execute Macro` enumerates the workbook's VBA modules ("VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']") | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (canonical "Cannot run the -macro" COMException) → `jobs logs` (VBA-module enumeration is the -smoking gun) → workflow source review (confirms configured -`MacroName: "RunImport"`) → conclude branch 1. - -> **Note on fixtures.** Synthetic. The macro names and module -> names are placeholders. The test grades whether the agent -> distinguishes branch 1 (macro absent) from branch 5 (macro -> present but Trust Center disabled) — the absence-from-enumeration -> evidence is decisive. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/abcc25d897.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/abcc25d897.json deleted file mode 100644 index a758c15b2f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/abcc25d897.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/ba5575ec73.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/ba5575ec73.json deleted file mode 100644 index b86b136a40..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/ba5575ec73.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelMacroProcess", - "Source": "Schedule", - "Reference": "HourlyMacroA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/e7e248f935.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/e7e248f935.json deleted file mode 100644 index 3541ccdb5e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/e7e248f935.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled.\r\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\r\n at UiPath.Excel.Activities.ExecuteMacro.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelMacroProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlyMacroA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "ActivityName": "ExecuteMacro_1", - "ActivityDisplayName": "Execute Macro: RunImport", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680099001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/manifest.json deleted file mode 100644 index 411c9939e9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Execute Macro scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "ba5575ec73.json"}, - {"match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", "file": "ba5575ec73.json"}, - {"match": "or jobs get \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "e7e248f935.json"}, - {"match": "or jobs get aa111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "e7e248f935.json"}, - {"match": "or jobs history \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "abcc25d897.json"}, - {"match": "or jobs history aa111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "abcc25d897.json"}, - {"match": "or jobs logs \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "3412bc7963.json"}, - {"match": "or jobs logs aa111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "3412bc7963.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..03200f9b89 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelMacroProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlyMacroA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled.\\r\\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\\r\\n at UiPath.Excel.Activities.ExecuteMacro.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelMacroProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlyMacroA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"ActivityName\": \"ExecuteMacro_1\",\r\n \"ActivityDisplayName\": \"Execute Macro: RunImport\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680099001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa1111cc-001\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-002\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelMacroProcess] Starting daily import — running macro RunImport on C:\\\\Robot\\\\Data\\\\sales-macro.xlsm\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-003\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-macro.xlsm — workbook opened\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-004\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.110Z\",\r\n \"Message\": \"Execute Macro: RunImport — enumerating VBA modules in workbook\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-005\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.310Z\",\r\n \"Message\": \"Execute Macro: RunImport — VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-006\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.510Z\",\r\n \"Message\": \"Execute Macro: RunImport — dispatching macro via COM (Application.Run 'RunImport')\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111cc-007\",\r\n \"JobKey\": \"aa111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Execute Macro: RunImport — System.Runtime.InteropServices.COMException: Cannot run the macro 'RunImport'. The macro may not be available in this workbook or all macros may be disabled.\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/task.yaml index ab0a9e1bb2..780f2a063e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-not-found/task.yaml @@ -24,21 +24,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/README.md new file mode 100644 index 0000000000..45a5f0b769 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/README.md @@ -0,0 +1,64 @@ +# Excel Execute Macro — Trust Center Blocks Macros On Robot Host + +This scenario reproduces an Execute Macro failure with the canonical +"Cannot run the macro..." COMException — same error string as the +macro-not-found case, but here the macro IS present in the +workbook. Excel's Trust Center policy on the Robot host's user +profile blocks macro execution. The job ends with: + +``` +System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled. +``` + +## What this scenario uncovers + +**Root Cause:** The macro `Module1.ProcessData` exists in the +workbook's VBA project (confirmed by the activity's VBA-module +enumeration in job logs). The error's wording covers two cases: +"macro not in workbook" (branch 1) OR "all macros disabled" +(branch 5) — both produce identical COMException text. Here, the +second case applies: Excel's Trust Center on the Robot host's user +profile is configured to disable macros without notification, and +the workbook is neither in a Trusted Location nor signed by a +trusted publisher. + +The user's "works on my dev machine" clue is the diagnostic: +developer Excel has permissive Trust Center defaults (or trusted +locations covering the dev workspace); Robot host's Excel under +the `UIPATH\AUTOMATION1` user does not. + +The agent must distinguish branch 5 from branch 1 by combining: +1. **VBA enumeration evidence** — job logs show the macro IS + present in the workbook (rules out branch 1). +2. **User clue** — "works on dev machine" is the canonical + per-host Trust Center divergence signal. + +CLI cannot inspect the Robot host's Trust Center settings; agent +must hand the user host-side inspection commands. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/execute-macro-failures.md` +(the "Macros disabled by Trust Center policy" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelMacroProcess` project — `Use Excel File` → `Execute Macro: Module1.ProcessData` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` ("Cannot run the macro" +COMException) → `jobs logs` (VBA enumeration confirms macro IS +present) → workflow source (confirms configured MacroName matches +enumeration entry) → conclude branch 5 (since macro presence rules +out branch 1) → recommend host-side Trust Center inspection. + +> **Note on fixtures.** Synthetic. The macro names and Robot user +> are placeholders representative of a real Trust Center +> deployment. The test grades whether the agent surfaces the +> Trust Center / Trusted Locations branch using the combination +> of VBA-enumeration evidence + the dev-machine user clue. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/50c35db7bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/50c35db7bb.json deleted file mode 100644 index c1f63a6106..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/50c35db7bb.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc3333cc-001", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333cc-002", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelMacroProcess] Starting daily import — running macro Module1.ProcessData on C:\\Robot\\Data\\sales-macro.xlsm", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333cc-003", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-macro.xlsm — workbook opened", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333cc-004", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.110Z", - "Message": "Execute Macro: Module1.ProcessData — enumerating VBA modules in workbook", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333cc-005", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.310Z", - "Message": "Execute Macro: Module1.ProcessData — VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333cc-006", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.510Z", - "Message": "Execute Macro: Module1.ProcessData — dispatching macro via COM (Application.Run 'Module1.ProcessData')", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333cc-007", - "JobKey": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Execute Macro: Module1.ProcessData — System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled.", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/96db710dff.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/96db710dff.json deleted file mode 100644 index 7c1f133275..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/96db710dff.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9cc33333-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled.\r\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\r\n at UiPath.Excel.Activities.ExecuteMacro.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelMacroProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlyMacroA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "ActivityName": "ExecuteMacro_1", - "ActivityDisplayName": "Execute Macro: Module1.ProcessData", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680099003 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/README.md deleted file mode 100644 index db4f6900fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# Excel Execute Macro — Trust Center Blocks Macros On Robot Host - -This scenario reproduces an Execute Macro failure with the canonical -"Cannot run the macro..." COMException — same error string as the -macro-not-found case, but here the macro IS present in the -workbook. Excel's Trust Center policy on the Robot host's user -profile blocks macro execution. The job ends with: - -``` -System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled. -``` - -## What this scenario uncovers - -**Root Cause:** The macro `Module1.ProcessData` exists in the -workbook's VBA project (confirmed by the activity's VBA-module -enumeration in job logs). The error's wording covers two cases: -"macro not in workbook" (branch 1) OR "all macros disabled" -(branch 5) — both produce identical COMException text. Here, the -second case applies: Excel's Trust Center on the Robot host's user -profile is configured to disable macros without notification, and -the workbook is neither in a Trusted Location nor signed by a -trusted publisher. - -The user's "works on my dev machine" clue is the diagnostic: -developer Excel has permissive Trust Center defaults (or trusted -locations covering the dev workspace); Robot host's Excel under -the `UIPATH\AUTOMATION1` user does not. - -The agent must distinguish branch 5 from branch 1 by combining: -1. **VBA enumeration evidence** — job logs show the macro IS - present in the workbook (rules out branch 1). -2. **User clue** — "works on dev machine" is the canonical - per-host Trust Center divergence signal. - -CLI cannot inspect the Robot host's Trust Center settings; agent -must hand the user host-side inspection commands. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/execute-macro-failures.md` -(the "Macros disabled by Trust Center policy" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelMacroProcess` project — `Use Excel File` → `Execute Macro: Module1.ProcessData` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entry shows the VBA enumeration includes `Module1.ProcessData` (the macro IS in the workbook) | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` ("Cannot run the macro" -COMException) → `jobs logs` (VBA enumeration confirms macro IS -present) → workflow source (confirms configured MacroName matches -enumeration entry) → conclude branch 5 (since macro presence rules -out branch 1) → recommend host-side Trust Center inspection. - -> **Note on fixtures.** Synthetic. The macro names and Robot user -> are placeholders representative of a real Trust Center -> deployment. The test grades whether the agent surfaces the -> Trust Center / Trusted Locations branch using the combination -> of VBA-enumeration evidence + the dev-machine user clue. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/c146d364e7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/c146d364e7.json deleted file mode 100644 index 6131889223..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/c146d364e7.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelMacroProcess", - "Source": "Schedule", - "Reference": "HourlyMacroA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/dc4905c809.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/dc4905c809.json deleted file mode 100644 index ec6289aee7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/dc4905c809.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "cc333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/manifest.json deleted file mode 100644 index 1bc642f04e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Execute Macro scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "c146d364e7.json"}, - {"match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", "file": "c146d364e7.json"}, - {"match": "or jobs get \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "96db710dff.json"}, - {"match": "or jobs get cc333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "96db710dff.json"}, - {"match": "or jobs history \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "dc4905c809.json"}, - {"match": "or jobs history cc333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "dc4905c809.json"}, - {"match": "or jobs logs \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "50c35db7bb.json"}, - {"match": "or jobs logs cc333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "50c35db7bb.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/uip-fixture.json new file mode 100644 index 0000000000..91636ff64d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelMacroProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlyMacroA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cc333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9cc33333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled.\\r\\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\\r\\n at UiPath.Excel.Activities.ExecuteMacro.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelMacroProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlyMacroA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"ActivityName\": \"ExecuteMacro_1\",\r\n \"ActivityDisplayName\": \"Execute Macro: Module1.ProcessData\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680099003\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cc333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cc333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc3333cc-001\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333cc-002\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelMacroProcess] Starting daily import — running macro Module1.ProcessData on C:\\\\Robot\\\\Data\\\\sales-macro.xlsm\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333cc-003\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-macro.xlsm — workbook opened\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333cc-004\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.110Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — enumerating VBA modules in workbook\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333cc-005\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.310Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333cc-006\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.510Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — dispatching macro via COM (Application.Run 'Module1.ProcessData')\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333cc-007\",\r\n \"JobKey\": \"cc333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — System.Runtime.InteropServices.COMException: Cannot run the macro 'Module1.ProcessData'. The macro may not be available in this workbook or all macros may be disabled.\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/task.yaml index 1d9166b7e8..43b33f3ecd 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-trust-center/task.yaml @@ -25,21 +25,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/README.md new file mode 100644 index 0000000000..ccf14e4357 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/README.md @@ -0,0 +1,49 @@ +# Excel Execute Macro — VBA Error Inside The Macro + +This scenario reproduces an Execute Macro failure where the macro +dispatched successfully but threw a VBA runtime error during +execution. The job ends with: + +``` +System.Runtime.InteropServices.COMException (0x80020009): Exception occurred. + ---> Run-time error '91': Object variable or With block variable not set +``` + +## What this scenario uncovers + +**Root Cause:** The macro `Module1.ProcessData` ran successfully past +the macro-lookup stage (Excel dispatched the call; VBA started +executing), then threw `Run-time error 91` inside the macro code. +HRESULT `0x80020009 DISP_E_EXCEPTION` is Excel's COM wrapper for +"the macro threw via IDispatch"; the inner VBA error is the actual +cause. Run-time error 91 specifically means the macro tried to use +an object variable that was never `Set` — typically a `Range`, +`Worksheet`, or `Workbook` reference sourced from a lookup that +returned `Nothing`. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/execute-macro-failures.md` +(the "VBA error inside the macro" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelMacroProcess` project — `Use Excel File` → `Execute Macro: Module1.ProcessData` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (HRESULT 0x80020009 + inner +Run-time error 91) → `jobs logs` (Trace shows macro dispatched, then +faulted with the inner error) → workflow source (Macro name + +configured properties) → conclude branch 2. + +> **Note on fixtures.** Synthetic. The macro name, module name, and +> specific Run-time error line in the logs are placeholders. The test +> grades whether the agent identifies the inner VBA error as the +> load-bearing detail (branch 2) and distinguishes it from branch 7 +> (which would have a different inner error: 424 Object Required or +> 429 ActiveX-cant-create). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/45349b92d2.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/45349b92d2.json deleted file mode 100644 index 027fa33a1e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/45349b92d2.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelMacroProcess", - "Source": "Schedule", - "Reference": "HourlyMacroA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.Runtime.InteropServices.COMException (0x80020009): Exception occurred. ---> Run-time error '91': Object variable or With block variable not set", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/5af43034d6.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/5af43034d6.json deleted file mode 100644 index 74a4d6d1e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/5af43034d6.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException (0x80020009): Exception occurred.\r\n ---> Run-time error '91': Object variable or With block variable not set\r\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\r\n at UiPath.Excel.Activities.ExecuteMacro.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelMacroProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlyMacroA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "HResult": "0x80020009", - "ActivityName": "ExecuteMacro_1", - "ActivityDisplayName": "Execute Macro: Module1.ProcessData", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680099002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/README.md deleted file mode 100644 index c23d94f874..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Excel Execute Macro — VBA Error Inside The Macro - -This scenario reproduces an Execute Macro failure where the macro -dispatched successfully but threw a VBA runtime error during -execution. The job ends with: - -``` -System.Runtime.InteropServices.COMException (0x80020009): Exception occurred. - ---> Run-time error '91': Object variable or With block variable not set -``` - -## What this scenario uncovers - -**Root Cause:** The macro `Module1.ProcessData` ran successfully past -the macro-lookup stage (Excel dispatched the call; VBA started -executing), then threw `Run-time error 91` inside the macro code. -HRESULT `0x80020009 DISP_E_EXCEPTION` is Excel's COM wrapper for -"the macro threw via IDispatch"; the inner VBA error is the actual -cause. Run-time error 91 specifically means the macro tried to use -an object variable that was never `Set` — typically a `Range`, -`Worksheet`, or `Workbook` reference sourced from a lookup that -returned `Nothing`. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/execute-macro-failures.md` -(the "VBA error inside the macro" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelMacroProcess` project — `Use Excel File` → `Execute Macro: Module1.ProcessData` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field includes the inner Run-time error 91 text | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (HRESULT 0x80020009 + inner -Run-time error 91) → `jobs logs` (Trace shows macro dispatched, then -faulted with the inner error) → workflow source (Macro name + -configured properties) → conclude branch 2. - -> **Note on fixtures.** Synthetic. The macro name, module name, and -> specific Run-time error line in the logs are placeholders. The test -> grades whether the agent identifies the inner VBA error as the -> load-bearing detail (branch 2) and distinguishes it from branch 7 -> (which would have a different inner error: 424 Object Required or -> 429 ActiveX-cant-create). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/b01622a007.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/b01622a007.json deleted file mode 100644 index 05f172e615..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/b01622a007.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-bb2222cc-001", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-002", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelMacroProcess] Starting daily import — running macro Module1.ProcessData on C:\\Robot\\Data\\sales-macro.xlsm", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-003", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-macro.xlsm — workbook opened", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-004", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.110Z", - "Message": "Execute Macro: Module1.ProcessData — enumerating VBA modules in workbook", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-005", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.310Z", - "Message": "Execute Macro: Module1.ProcessData — VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-006", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.412Z", - "Message": "Execute Macro: Module1.ProcessData — dispatching macro via COM (Application.Run 'Module1.ProcessData')", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-007", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.612Z", - "Message": "Execute Macro: Module1.ProcessData — macro started executing (Application.Run returned dispatch handle; VBA in flight)", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222cc-008", - "JobKey": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Execute Macro: Module1.ProcessData — System.Runtime.InteropServices.COMException (0x80020009): Exception occurred. ---> Run-time error '91': Object variable or With block variable not set", - "ProcessName": "ExcelMacroProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/b30bd8012a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/b30bd8012a.json deleted file mode 100644 index 3473773064..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/b30bd8012a.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "bb222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/manifest.json deleted file mode 100644 index 2d310ebeab..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Execute Macro scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "45349b92d2.json"}, - {"match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", "file": "45349b92d2.json"}, - {"match": "or jobs get \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "5af43034d6.json"}, - {"match": "or jobs get bb222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "5af43034d6.json"}, - {"match": "or jobs history \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "b30bd8012a.json"}, - {"match": "or jobs history bb222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "b30bd8012a.json"}, - {"match": "or jobs logs \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "b01622a007.json"}, - {"match": "or jobs logs bb222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "b01622a007.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/uip-fixture.json new file mode 100644 index 0000000000..d6e0cb6875 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelMacroProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlyMacroA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x80020009): Exception occurred. ---> Run-time error '91': Object variable or With block variable not set\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x80020009): Exception occurred.\\r\\n ---> Run-time error '91': Object variable or With block variable not set\\r\\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\\r\\n at UiPath.Excel.Activities.ExecuteMacro.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelMacroProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlyMacroA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"HResult\": \"0x80020009\",\r\n \"ActivityName\": \"ExecuteMacro_1\",\r\n \"ActivityDisplayName\": \"Execute Macro: Module1.ProcessData\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680099002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "bb222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-bb2222cc-001\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-002\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelMacroProcess] Starting daily import — running macro Module1.ProcessData on C:\\\\Robot\\\\Data\\\\sales-macro.xlsm\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-003\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-macro.xlsm — workbook opened\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-004\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.110Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — enumerating VBA modules in workbook\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-005\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.310Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — VBA modules: ['Module1.ProcessData', 'Module2.UpdateReport']\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-006\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.412Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — dispatching macro via COM (Application.Run 'Module1.ProcessData')\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-007\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.612Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — macro started executing (Application.Run returned dispatch handle; VBA in flight)\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222cc-008\",\r\n \"JobKey\": \"bb222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Execute Macro: Module1.ProcessData — System.Runtime.InteropServices.COMException (0x80020009): Exception occurred. ---> Run-time error '91': Object variable or With block variable not set\",\r\n \"ProcessName\": \"ExcelMacroProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/task.yaml index ccfb2e678b..99f6403630 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-macro-vba-error/task.yaml @@ -25,21 +25,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/README.md new file mode 100644 index 0000000000..a9992476e0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/README.md @@ -0,0 +1,47 @@ +# Excel Read Range — File Locked by Concurrent Robot Jobs + +This scenario reproduces an `ExcelReadRange` failure caused by **two +Orchestrator jobs racing against the same workbook on the same host**. +The job ends with: + +``` +System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. +``` + +## What this scenario uncovers + +**Root Cause:** Job `ExcelDailyImport` (key `aa111111-...`) and a sibling +job (key `bb222222-...`) ran the same process against the same workbook +on overlapping schedules. The first job to start acquired the file lock +on `sales-2026-05.xlsx`; the second job's Read Range hit `IOException` +because the OS refused the concurrent open. Two triggers (`HourlySalesA` +and `HourlySalesB`) fire the same process every 30 minutes, offset by +0 minutes — their Running windows always overlap. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/workbook-file-locked.md` +(the "Concurrent Robot jobs racing on the same workbook" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal `ExcelDailyImport` project — `Use Excel File` → `Read Range` on `sales-2026-05.xlsx` | + +The fixture set models the agent's expected investigation chain: +`folders list-current-user` → `jobs list --state Faulted` → `jobs get` +→ `jobs logs` → `jobs list` (broader, shows the sibling job) → `jobs +get ` → `triggers list` (shows overlap). The sibling job +key, its Running window, and the two-trigger overlap pattern together +identify the concurrent-jobs branch. + +> **Note on fixtures.** Fixtures are synthetic, authored from the +> documented playbook signature rather than captured from a real +> `.investigation/` session. The job keys, folder key, workbook path, +> and trigger names are placeholders — the test grades whether the +> agent surfaces the concurrent-jobs branch with evidence (sibling job +> key + trigger overlap), not the specific names. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/23524abd8d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/23524abd8d.json deleted file mode 100644 index 2f5fc08c80..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/23524abd8d.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-2222-3333-4444-555566667777", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'.\r\n ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.\r\n at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)\r\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecuteAsync(NativeActivityContext context)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecuteAsync(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-19T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesB", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:01.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "ApplicationException", - "ActivityName": "ExcelApplicationScope_1", - "ActivityDisplayName": "Use Excel File: sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-2222-3333-4444-555566667777", - "Id": 680011001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/8c7e2bc2e9.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/8c7e2bc2e9.json deleted file mode 100644 index 81e0ec4f33..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/8c7e2bc2e9.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "aa111111-2222-3333-4444-555566667777", - "State": "Pending", - "Timestamp": "2026-05-19T08:00:01.005Z" - }, - { - "Key": "aa111111-2222-3333-4444-555566667777", - "State": "Running", - "Timestamp": "2026-05-19T08:00:01.300Z" - }, - { - "Key": "aa111111-2222-3333-4444-555566667777", - "State": "Faulted", - "Timestamp": "2026-05-19T08:00:01.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/README.md deleted file mode 100644 index 82ccbe9248..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Excel Read Range — File Locked by Concurrent Robot Jobs - -This scenario reproduces an `ExcelReadRange` failure caused by **two -Orchestrator jobs racing against the same workbook on the same host**. -The job ends with: - -``` -System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. -``` - -## What this scenario uncovers - -**Root Cause:** Job `ExcelDailyImport` (key `aa111111-...`) and a sibling -job (key `bb222222-...`) ran the same process against the same workbook -on overlapping schedules. The first job to start acquired the file lock -on `sales-2026-05.xlsx`; the second job's Read Range hit `IOException` -because the OS refused the concurrent open. Two triggers (`HourlySalesA` -and `HourlySalesB`) fire the same process every 30 minutes, offset by -0 minutes — their Running windows always overlap. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/workbook-file-locked.md` -(the "Concurrent Robot jobs racing on the same workbook" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | minimal `ExcelDailyImport` project — `Use Excel File` → `Read Range` on `sales-2026-05.xlsx` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The fixture set models the agent's expected investigation chain: -`folders list-current-user` → `jobs list --state Faulted` → `jobs get` -→ `jobs logs` → `jobs list` (broader, shows the sibling job) → `jobs -get ` → `triggers list` (shows overlap). The sibling job -key, its Running window, and the two-trigger overlap pattern together -identify the concurrent-jobs branch. - -> **Note on fixtures.** Fixtures are synthetic, authored from the -> documented playbook signature rather than captured from a real -> `.investigation/` session. The job keys, folder key, workbook path, -> and trigger names are placeholders — the test grades whether the -> agent surfaces the concurrent-jobs branch with evidence (sibling job -> key + trigger overlap), not the specific names. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/adb975c3e4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/adb975c3e4.json deleted file mode 100644 index 0d773a7af6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/adb975c3e4.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-2222-3333-4444-555566667777", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesB", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/af3254f5e8.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/af3254f5e8.json deleted file mode 100644 index d22e574d57..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/af3254f5e8.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa111111-001", - "JobKey": "aa111111-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-002", - "JobKey": "aa111111-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-003", - "JobKey": "aa111111-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.512Z", - "Message": "Use Excel File: sales-2026-05.xlsx — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-004", - "JobKey": "aa111111-2222-3333-4444-555566667777", - "Level": "Error", - "Timestamp": "2026-05-19T08:00:01.802Z", - "Message": "Use Excel File: sales-2026-05.xlsx — UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/b51fa13336.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/b51fa13336.json deleted file mode 100644 index 54813ec54b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/b51fa13336.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-2222-3333-4444-555566667777", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesB", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - }, - { - "Key": "bb222222-3333-4444-5555-666677778888", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:00:08.105Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - }, - { - "Key": "cc333333-1111-2222-3333-444455556666", - "StartTime": "2026-05-19T07:30:00.500Z", - "EndTime": "2026-05-19T07:30:07.205Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - }, - { - "Key": "dd444444-1111-2222-3333-444455556666", - "StartTime": "2026-05-19T07:30:01.300Z", - "EndTime": "2026-05-19T07:30:01.792Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesB", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/b9dded45dd.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/b9dded45dd.json deleted file mode 100644 index 32d5131718..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/b9dded45dd.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-3333-4444-5555-666677778888", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:00:08.105Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-3333-4444-5555-666677778888", - "Info": null, - "CreationTime": "2026-05-19T08:00:00.205Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:08.105Z", - "JobError": {}, - "ErrorCode": null, - "TraceId": "bb222222-3333-4444-5555-666677778888", - "Id": 680011002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/e2c7d9aa71.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/e2c7d9aa71.json deleted file mode 100644 index fe6c06f48d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/e2c7d9aa71.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "TriggersList", - "Data": [ - { - "Key": "t1111111-aaaa-bbbb-cccc-ddddeeeeffff", - "Name": "HourlySalesA", - "Enabled": true, - "ProcessKey": "5680101", - "ProcessName": "ExcelDailyImport", - "Cron": "0 0/30 * * * ?", - "CronSummary": "Every 30 minutes", - "TimeZoneId": "UTC", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "FolderName": "ExcelImports", - "StartProcessCron": "0 0/30 * * * ?", - "StartProcessCronSummary": "Every 30 minutes", - "StartProcessCronDetails": "Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z", - "JobsCount": 1, - "RuntimeType": "Unattended" - }, - { - "Key": "t2222222-aaaa-bbbb-cccc-ddddeeeeffff", - "Name": "HourlySalesB", - "Enabled": true, - "ProcessKey": "5680101", - "ProcessName": "ExcelDailyImport", - "Cron": "0 0/30 * * * ?", - "CronSummary": "Every 30 minutes", - "TimeZoneId": "UTC", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "FolderName": "ExcelImports", - "StartProcessCron": "0 0/30 * * * ?", - "StartProcessCronSummary": "Every 30 minutes", - "StartProcessCronDetails": "Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z", - "JobsCount": 1, - "RuntimeType": "Unattended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/manifest.json deleted file mode 100644 index 21bc2624f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/m/r/manifest.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "adb975c3e4.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "adb975c3e4.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "b51fa13336.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "b51fa13336.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --process-name \"ExcelDailyImport\" --output json", - "file": "b51fa13336.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --process-name ExcelDailyImport --output json", - "file": "b51fa13336.json" - }, - { - "match": "or jobs get \"aa111111-2222-3333-4444-555566667777\" --output json", - "file": "23524abd8d.json" - }, - { - "match": "or jobs get aa111111-2222-3333-4444-555566667777 --output json", - "file": "23524abd8d.json" - }, - { - "match": "or jobs get \"bb222222-3333-4444-5555-666677778888\" --output json", - "file": "b9dded45dd.json" - }, - { - "match": "or jobs get bb222222-3333-4444-5555-666677778888 --output json", - "file": "b9dded45dd.json" - }, - { - "match": "or jobs history \"aa111111-2222-3333-4444-555566667777\" --output json", - "file": "8c7e2bc2e9.json" - }, - { - "match": "or jobs history aa111111-2222-3333-4444-555566667777 --output json", - "file": "8c7e2bc2e9.json" - }, - { - "match": "or jobs logs \"aa111111-2222-3333-4444-555566667777\" --output json", - "file": "af3254f5e8.json" - }, - { - "match": "or jobs logs aa111111-2222-3333-4444-555566667777 --output json", - "file": "af3254f5e8.json" - }, - { - "match": "or triggers list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "e2c7d9aa71.json" - }, - { - "match": "or triggers list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "e2c7d9aa71.json" - }, - { - "match": "or triggers list --folder-path \"ExcelImports\" --output json", - "file": "e2c7d9aa71.json" - }, - { - "match": "or triggers list --folder-path ExcelImports --output json", - "file": "e2c7d9aa71.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/uip-fixture.json new file mode 100644 index 0000000000..003c6f2099 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/data/uip-fixture.json @@ -0,0 +1,147 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesB\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--process-name", + "ExcelDailyImport", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesB\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"bb222222-3333-4444-5555-666677778888\",\r\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T08:00:08.105Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"cc333333-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:30:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:30:07.205Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"dd444444-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:30:01.300Z\",\r\n \"EndTime\": \"2026-05-19T07:30:01.792Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesB\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesB\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"bb222222-3333-4444-5555-666677778888\",\r\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T08:00:08.105Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"cc333333-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:30:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:30:07.205Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"dd444444-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:30:01.300Z\",\r\n \"EndTime\": \"2026-05-19T07:30:01.792Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesB\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "triggers", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TriggersList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"t1111111-aaaa-bbbb-cccc-ddddeeeeffff\",\r\n \"Name\": \"HourlySalesA\",\r\n \"Enabled\": true,\r\n \"ProcessKey\": \"5680101\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"Cron\": \"0 0/30 * * * ?\",\r\n \"CronSummary\": \"Every 30 minutes\",\r\n \"TimeZoneId\": \"UTC\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"FolderName\": \"ExcelImports\",\r\n \"StartProcessCron\": \"0 0/30 * * * ?\",\r\n \"StartProcessCronSummary\": \"Every 30 minutes\",\r\n \"StartProcessCronDetails\": \"Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z\",\r\n \"JobsCount\": 1,\r\n \"RuntimeType\": \"Unattended\"\r\n },\r\n {\r\n \"Key\": \"t2222222-aaaa-bbbb-cccc-ddddeeeeffff\",\r\n \"Name\": \"HourlySalesB\",\r\n \"Enabled\": true,\r\n \"ProcessKey\": \"5680101\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"Cron\": \"0 0/30 * * * ?\",\r\n \"CronSummary\": \"Every 30 minutes\",\r\n \"TimeZoneId\": \"UTC\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"FolderName\": \"ExcelImports\",\r\n \"StartProcessCron\": \"0 0/30 * * * ?\",\r\n \"StartProcessCronSummary\": \"Every 30 minutes\",\r\n \"StartProcessCronDetails\": \"Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z\",\r\n \"JobsCount\": 1,\r\n \"RuntimeType\": \"Unattended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "triggers", + "list", + "--folder-path", + "ExcelImports", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TriggersList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"t1111111-aaaa-bbbb-cccc-ddddeeeeffff\",\r\n \"Name\": \"HourlySalesA\",\r\n \"Enabled\": true,\r\n \"ProcessKey\": \"5680101\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"Cron\": \"0 0/30 * * * ?\",\r\n \"CronSummary\": \"Every 30 minutes\",\r\n \"TimeZoneId\": \"UTC\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"FolderName\": \"ExcelImports\",\r\n \"StartProcessCron\": \"0 0/30 * * * ?\",\r\n \"StartProcessCronSummary\": \"Every 30 minutes\",\r\n \"StartProcessCronDetails\": \"Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z\",\r\n \"JobsCount\": 1,\r\n \"RuntimeType\": \"Unattended\"\r\n },\r\n {\r\n \"Key\": \"t2222222-aaaa-bbbb-cccc-ddddeeeeffff\",\r\n \"Name\": \"HourlySalesB\",\r\n \"Enabled\": true,\r\n \"ProcessKey\": \"5680101\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"Cron\": \"0 0/30 * * * ?\",\r\n \"CronSummary\": \"Every 30 minutes\",\r\n \"TimeZoneId\": \"UTC\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"FolderName\": \"ExcelImports\",\r\n \"StartProcessCron\": \"0 0/30 * * * ?\",\r\n \"StartProcessCronSummary\": \"Every 30 minutes\",\r\n \"StartProcessCronDetails\": \"Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z\",\r\n \"JobsCount\": 1,\r\n \"RuntimeType\": \"Unattended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa111111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\\r\\n ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\\r\\n at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)\\r\\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecuteAsync(NativeActivityContext context)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecuteAsync(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesB\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"ApplicationException\",\r\n \"ActivityName\": \"ExcelApplicationScope_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"Id\": 680011001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb222222-3333-4444-5555-666677778888", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-3333-4444-5555-666677778888\",\r\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T08:00:08.105Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-3333-4444-5555-666677778888\",\r\n \"Info\": null,\r\n \"CreationTime\": \"2026-05-19T08:00:00.205Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:08.105Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": null,\r\n \"TraceId\": \"bb222222-3333-4444-5555-666677778888\",\r\n \"Id\": 680011002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa111111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa111111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa111111-001\",\r\n \"JobKey\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-002\",\r\n \"JobKey\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-003\",\r\n \"JobKey\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.512Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-004\",\r\n \"JobKey\": \"aa111111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.802Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/task.yaml index 2d19db912d..1baa46a22c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-concurrent-jobs/task.yaml @@ -23,21 +23,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/README.md new file mode 100644 index 0000000000..1ffaff9ecc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/README.md @@ -0,0 +1,47 @@ +# Excel Read Range — Workbook File Deleted Upstream + +This scenario reproduces a Read Range failure where the configured +`WorkbookPath` is a stable absolute literal that has worked for weeks, +but the file has now been deleted or moved by an upstream process. The +job ends with: + +``` +System.IO.FileNotFoundException: Could not find file 'C:\Robot\Data\sales-2026-05.xlsx'. +``` + +## What this scenario uncovers + +**Root Cause:** The workbook +`C:\Robot\Data\sales-2026-05.xlsx` was deleted from the Robot host +between the prior (successful) run and the current (failed) run. The +configured `WorkbookPath` is an absolute literal that has not changed; +the file simply went missing. The recent-jobs list shows a clear +pattern — multiple successful runs followed by a single failed run on +the same path — that rules out a stable configuration problem and +points directly at branch 1 of the playbook. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-file-not-found.md` +(the "File moved or deleted upstream" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` with absolute literal `WorkbookPath: "C:\Robot\Data\sales-2026-05.xlsx"`, then `Read Range` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (current, FileNotFoundException) +→ `jobs logs` (last activity log shows the open attempt) → `jobs list` +(broader; surfaces multiple prior **Successful** runs against the same +process / same path) → workflow source review (absolute literal path, +no relative / drive-letter / UNC concerns). + +> **Note on fixtures.** Synthetic. The job keys, folder key, and +> workbook path are placeholders — the test grades whether the agent +> surfaces the file-deleted-upstream branch using the prior-success +> pattern as evidence. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/160d07ab53.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/160d07ab53.json deleted file mode 100644 index a52222cc3d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/160d07ab53.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-8888-9999-aaaa-bbbbccccdddd", - "Info": "System.IO.FileNotFoundException: Could not find file 'C:\\Robot\\Data\\sales-2026-05.xlsx'.\r\n at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)\r\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\r\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:01.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "FileNotFoundException", - "ActivityName": "UseExcelFile_1", - "ActivityDisplayName": "Use Excel File: sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "Id": 680077001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/67562150ed.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/67562150ed.json deleted file mode 100644 index 25894b14e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/67562150ed.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "State": "Pending", - "Timestamp": "2026-05-20T08:00:01.005Z" - }, - { - "Key": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "State": "Running", - "Timestamp": "2026-05-20T08:00:01.300Z" - }, - { - "Key": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "State": "Faulted", - "Timestamp": "2026-05-20T08:00:01.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/6bb9f5d750.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/6bb9f5d750.json deleted file mode 100644 index 338293234b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/6bb9f5d750.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.IO.FileNotFoundException: Could not find file 'C:\\Robot\\Data\\sales-2026-05.xlsx'.", - "ErrorCode": "Activity" - }, - { - "Key": "bb222222-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T07:00:00.500Z", - "EndTime": "2026-05-20T07:00:08.105Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - }, - { - "Key": "cc333333-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T06:00:00.500Z", - "EndTime": "2026-05-20T06:00:07.205Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - }, - { - "Key": "dd444444-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T05:00:00.500Z", - "EndTime": "2026-05-20T05:00:06.412Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/README.md deleted file mode 100644 index fa3b1c1a74..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Excel Read Range — Workbook File Deleted Upstream - -This scenario reproduces a Read Range failure where the configured -`WorkbookPath` is a stable absolute literal that has worked for weeks, -but the file has now been deleted or moved by an upstream process. The -job ends with: - -``` -System.IO.FileNotFoundException: Could not find file 'C:\Robot\Data\sales-2026-05.xlsx'. -``` - -## What this scenario uncovers - -**Root Cause:** The workbook -`C:\Robot\Data\sales-2026-05.xlsx` was deleted from the Robot host -between the prior (successful) run and the current (failed) run. The -configured `WorkbookPath` is an absolute literal that has not changed; -the file simply went missing. The recent-jobs list shows a clear -pattern — multiple successful runs followed by a single failed run on -the same path — that rules out a stable configuration problem and -points directly at branch 1 of the playbook. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-file-not-found.md` -(the "File moved or deleted upstream" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` with absolute literal `WorkbookPath: "C:\Robot\Data\sales-2026-05.xlsx"`, then `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (current, FileNotFoundException) -→ `jobs logs` (last activity log shows the open attempt) → `jobs list` -(broader; surfaces multiple prior **Successful** runs against the same -process / same path) → workflow source review (absolute literal path, -no relative / drive-letter / UNC concerns). - -> **Note on fixtures.** Synthetic. The job keys, folder key, and -> workbook path are placeholders — the test grades whether the agent -> surfaces the file-deleted-upstream branch using the prior-success -> pattern as evidence. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/ad3693385b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/ad3693385b.json deleted file mode 100644 index 603ec097af..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/ad3693385b.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.IO.FileNotFoundException: Could not find file 'C:\\Robot\\Data\\sales-2026-05.xlsx'.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/ba3719a305.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/ba3719a305.json deleted file mode 100644 index e26311ccfd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/ba3719a305.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-8888-9999-aaaa-bbbbccccdddd", - "StartTime": "2026-05-20T07:00:00.500Z", - "EndTime": "2026-05-20T07:00:08.105Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-8888-9999-aaaa-bbbbccccdddd", - "Info": null, - "CreationTime": "2026-05-20T07:00:00.205Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T07:00:08.105Z", - "JobError": {}, - "ErrorCode": null, - "TraceId": "bb222222-8888-9999-aaaa-bbbbccccdddd", - "Id": 680077002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/f9dc038676.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/f9dc038676.json deleted file mode 100644 index 88361a5359..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/f9dc038676.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa111188-001", - "JobKey": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111188-002", - "JobKey": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111188-003", - "JobKey": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — opening workbook (WorkbookPath='C:\\Robot\\Data\\sales-2026-05.xlsx')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111188-004", - "JobKey": "aa111111-8888-9999-aaaa-bbbbccccdddd", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:01.802Z", - "Message": "Use Excel File: sales-2026-05.xlsx — System.IO.FileNotFoundException: Could not find file 'C:\\Robot\\Data\\sales-2026-05.xlsx'.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/manifest.json deleted file mode 100644 index 143c0fcf6a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/m/r/manifest.json +++ /dev/null @@ -1,70 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "ad3693385b.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "ad3693385b.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "6bb9f5d750.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "6bb9f5d750.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --process-name \"ExcelDailyImport\" --output json", - "file": "6bb9f5d750.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --process-name ExcelDailyImport --output json", - "file": "6bb9f5d750.json" - }, - { - "match": "or jobs get \"aa111111-8888-9999-aaaa-bbbbccccdddd\" --output json", - "file": "160d07ab53.json" - }, - { - "match": "or jobs get aa111111-8888-9999-aaaa-bbbbccccdddd --output json", - "file": "160d07ab53.json" - }, - { - "match": "or jobs get \"bb222222-8888-9999-aaaa-bbbbccccdddd\" --output json", - "file": "ba3719a305.json" - }, - { - "match": "or jobs get bb222222-8888-9999-aaaa-bbbbccccdddd --output json", - "file": "ba3719a305.json" - }, - { - "match": "or jobs history \"aa111111-8888-9999-aaaa-bbbbccccdddd\" --output json", - "file": "67562150ed.json" - }, - { - "match": "or jobs history aa111111-8888-9999-aaaa-bbbbccccdddd --output json", - "file": "67562150ed.json" - }, - { - "match": "or jobs logs \"aa111111-8888-9999-aaaa-bbbbccccdddd\" --output json", - "file": "f9dc038676.json" - }, - { - "match": "or jobs logs aa111111-8888-9999-aaaa-bbbbccccdddd --output json", - "file": "f9dc038676.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/uip-fixture.json new file mode 100644 index 0000000000..682535d8da --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/data/uip-fixture.json @@ -0,0 +1,119 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.FileNotFoundException: Could not find file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--process-name", + "ExcelDailyImport", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.FileNotFoundException: Could not find file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"bb222222-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T07:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T07:00:08.105Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"cc333333-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T06:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T06:00:07.205Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"dd444444-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T05:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T05:00:06.412Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.FileNotFoundException: Could not find file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"bb222222-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T07:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T07:00:08.105Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"cc333333-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T06:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T06:00:07.205Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"dd444444-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T05:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T05:00:06.412Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa111111-8888-9999-aaaa-bbbbccccdddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Info\": \"System.IO.FileNotFoundException: Could not find file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\\r\\n at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)\\r\\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\\r\\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"FileNotFoundException\",\r\n \"ActivityName\": \"UseExcelFile_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Id\": 680077001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb222222-8888-9999-aaaa-bbbbccccdddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-8888-9999-aaaa-bbbbccccdddd\",\r\n \"StartTime\": \"2026-05-20T07:00:00.500Z\",\r\n \"EndTime\": \"2026-05-20T07:00:08.105Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Info\": null,\r\n \"CreationTime\": \"2026-05-20T07:00:00.205Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T07:00:08.105Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": null,\r\n \"TraceId\": \"bb222222-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Id\": 680077002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa111111-8888-9999-aaaa-bbbbccccdddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa111111-8888-9999-aaaa-bbbbccccdddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa111188-001\",\r\n \"JobKey\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111188-002\",\r\n \"JobKey\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111188-003\",\r\n \"JobKey\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — opening workbook (WorkbookPath='C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111188-004\",\r\n \"JobKey\": \"aa111111-8888-9999-aaaa-bbbbccccdddd\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.802Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — System.IO.FileNotFoundException: Could not find file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/task.yaml index fc4d985e7b..5751ca127b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-file-deleted/task.yaml @@ -24,21 +24,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/README.md new file mode 100644 index 0000000000..ab4fef2ddd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/README.md @@ -0,0 +1,66 @@ +# Excel Read Range — File Locked, CLI Evidence Insufficient + +This scenario reproduces an `ExcelReadRange` failure where the workbook +is locked by SOMETHING, but the `uip` CLI evidence does NOT pinpoint +which cause-branch applies. The agent must recognize the boundary of +CLI evidence and recommend a host-side investigation rather than +guessing. + +The job ends with: + +``` +System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. +``` + +## What this scenario uncovers + +**Expected outcome:** The agent matches the +`activity-packages/excel-activities/playbooks/workbook-file-locked.md` +playbook AND tells the user the lock holder cannot be identified +through Orchestrator alone — they need to capture host-side evidence +(`Get-Process EXCEL`, Sysinternals `handle.exe -a `, and +`Get-SmbOpenFile` if the workbook is on a UNC path). + +**The branches that COULD apply** without further evidence: + +- User opened the workbook in Excel UI (branch 1) +- Hidden Excel instance under a different user session (branch 6) +- Network-share lock from a different host (branch 3) — only if the + workbook is on a UNC path +- Antivirus / EDR / Windows Search / sync client transient hold + (branch 5) + +**The branches that are ruled OUT by the CLI evidence:** + +- Concurrent Robot jobs (branch 4) — `or jobs list` shows no + concurrent sibling and `or triggers list` shows a single non- + overlapping trigger. +- Orphan EXCEL.EXE from a prior run of this workflow (branch 2) — no + prior faulted runs in the recent window, and `process/Main.xaml` + has no `ContinueOnError` on the scope and no swallowing TryCatch. + +The agent must NOT guess between the remaining branches without +host-side evidence. Confident guessing is the failure mode this test +catches. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project where `Main.xaml` is clean — no `ContinueOnError`, no swallowing TryCatch, modern `Use Excel File` scope | + +The expected investigation chain: `folders list-current-user` → `jobs +list --state Faulted` → `jobs get` → `jobs logs` → `jobs list` +(broader, returns only this single failure, no siblings, no prior +fails) → `triggers list` (single trigger) → workflow source review +(clean) → STOP. At this point CLI evidence is exhausted. The agent +must explicitly name the host-side commands and ask the user to run +them. + +> **Note on fixtures.** Synthetic. The job key, folder key, and +> workbook path are placeholders. What the test grades is the agent's +> awareness of evidence-availability boundaries. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/2fe5edebee.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/2fe5edebee.json deleted file mode 100644 index 87e2b593dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/2fe5edebee.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ee555555-001", - "JobKey": "ee555555-6666-7777-8888-99990000aaaa", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee555555-002", - "JobKey": "ee555555-6666-7777-8888-99990000aaaa", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee555555-003", - "JobKey": "ee555555-6666-7777-8888-99990000aaaa", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.512Z", - "Message": "Use Excel File: sales-2026-05.xlsx — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee555555-004", - "JobKey": "ee555555-6666-7777-8888-99990000aaaa", - "Level": "Error", - "Timestamp": "2026-05-19T08:00:01.802Z", - "Message": "Use Excel File: sales-2026-05.xlsx — UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/51183f1459.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/51183f1459.json deleted file mode 100644 index 0fe891a1ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/51183f1459.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ee555555-6666-7777-8888-99990000aaaa", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ee55555-6666-7777-8888-99990000aaaa", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'.\r\n ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.\r\n at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)\r\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\r\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-19T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:01.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "ApplicationException", - "ActivityName": "UseExcelFile_1", - "ActivityDisplayName": "Use Excel File: sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ee555555-6666-7777-8888-99990000aaaa", - "Id": 680033001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/5964b42a2b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/5964b42a2b.json deleted file mode 100644 index 4f2c7c082e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/5964b42a2b.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "ee555555-6666-7777-8888-99990000aaaa", - "State": "Pending", - "Timestamp": "2026-05-19T08:00:01.005Z" - }, - { - "Key": "ee555555-6666-7777-8888-99990000aaaa", - "State": "Running", - "Timestamp": "2026-05-19T08:00:01.300Z" - }, - { - "Key": "ee555555-6666-7777-8888-99990000aaaa", - "State": "Faulted", - "Timestamp": "2026-05-19T08:00:01.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/8d61cefe8e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/8d61cefe8e.json deleted file mode 100644 index 414a740444..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/8d61cefe8e.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ee555555-6666-7777-8888-99990000aaaa", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/README.md deleted file mode 100644 index af0ed2f23e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/README.md +++ /dev/null @@ -1,65 +0,0 @@ -# Excel Read Range — File Locked, CLI Evidence Insufficient - -This scenario reproduces an `ExcelReadRange` failure where the workbook -is locked by SOMETHING, but the `uip` CLI evidence does NOT pinpoint -which cause-branch applies. The agent must recognize the boundary of -CLI evidence and recommend a host-side investigation rather than -guessing. - -The job ends with: - -``` -System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. -``` - -## What this scenario uncovers - -**Expected outcome:** The agent matches the -`activity-packages/excel-activities/playbooks/workbook-file-locked.md` -playbook AND tells the user the lock holder cannot be identified -through Orchestrator alone — they need to capture host-side evidence -(`Get-Process EXCEL`, Sysinternals `handle.exe -a `, and -`Get-SmbOpenFile` if the workbook is on a UNC path). - -**The branches that COULD apply** without further evidence: - -- User opened the workbook in Excel UI (branch 1) -- Hidden Excel instance under a different user session (branch 6) -- Network-share lock from a different host (branch 3) — only if the - workbook is on a UNC path -- Antivirus / EDR / Windows Search / sync client transient hold - (branch 5) - -**The branches that are ruled OUT by the CLI evidence:** - -- Concurrent Robot jobs (branch 4) — `or jobs list` shows no - concurrent sibling and `or triggers list` shows a single non- - overlapping trigger. -- Orphan EXCEL.EXE from a prior run of this workflow (branch 2) — no - prior faulted runs in the recent window, and `process/Main.xaml` - has no `ContinueOnError` on the scope and no swallowing TryCatch. - -The agent must NOT guess between the remaining branches without -host-side evidence. Confident guessing is the failure mode this test -catches. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | `ExcelDailyImport` project where `Main.xaml` is clean — no `ContinueOnError`, no swallowing TryCatch, modern `Use Excel File` scope | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored so the agent can rule out concurrent / orphan branches but cannot identify a positive cause | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The expected investigation chain: `folders list-current-user` → `jobs -list --state Faulted` → `jobs get` → `jobs logs` → `jobs list` -(broader, returns only this single failure, no siblings, no prior -fails) → `triggers list` (single trigger) → workflow source review -(clean) → STOP. At this point CLI evidence is exhausted. The agent -must explicitly name the host-side commands and ask the user to run -them. - -> **Note on fixtures.** Synthetic. The job key, folder key, and -> workbook path are placeholders. What the test grades is the agent's -> awareness of evidence-availability boundaries. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/d1082fb9ed.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/d1082fb9ed.json deleted file mode 100644 index f77898b138..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/d1082fb9ed.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "TriggersList", - "Data": [ - { - "Key": "t1111111-aaaa-bbbb-cccc-ddddeeeeffff", - "Name": "HourlySalesA", - "Enabled": true, - "ProcessKey": "5680101", - "ProcessName": "ExcelDailyImport", - "Cron": "0 0/30 * * * ?", - "CronSummary": "Every 30 minutes", - "TimeZoneId": "UTC", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "FolderName": "ExcelImports", - "StartProcessCron": "0 0/30 * * * ?", - "StartProcessCronSummary": "Every 30 minutes", - "StartProcessCronDetails": "Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z", - "JobsCount": 1, - "RuntimeType": "Unattended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/fe864065f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/fe864065f5.json deleted file mode 100644 index 177d2e14dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/fe864065f5.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ee555555-6666-7777-8888-99990000aaaa", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - }, - { - "Key": "ff666666-1111-2222-3333-444455556666", - "StartTime": "2026-05-19T07:30:00.500Z", - "EndTime": "2026-05-19T07:30:06.700Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - }, - { - "Key": "ff777777-1111-2222-3333-444455556666", - "StartTime": "2026-05-19T07:00:00.500Z", - "EndTime": "2026-05-19T07:00:05.900Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/manifest.json deleted file mode 100644 index d33463718c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/m/r/manifest.json +++ /dev/null @@ -1,70 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "8d61cefe8e.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "8d61cefe8e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "fe864065f5.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "fe864065f5.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --process-name \"ExcelDailyImport\" --output json", - "file": "fe864065f5.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --process-name ExcelDailyImport --output json", - "file": "fe864065f5.json" - }, - { - "match": "or jobs get \"ee555555-6666-7777-8888-99990000aaaa\" --output json", - "file": "51183f1459.json" - }, - { - "match": "or jobs get ee555555-6666-7777-8888-99990000aaaa --output json", - "file": "51183f1459.json" - }, - { - "match": "or jobs history \"ee555555-6666-7777-8888-99990000aaaa\" --output json", - "file": "5964b42a2b.json" - }, - { - "match": "or jobs history ee555555-6666-7777-8888-99990000aaaa --output json", - "file": "5964b42a2b.json" - }, - { - "match": "or jobs logs \"ee555555-6666-7777-8888-99990000aaaa\" --output json", - "file": "2fe5edebee.json" - }, - { - "match": "or jobs logs ee555555-6666-7777-8888-99990000aaaa --output json", - "file": "2fe5edebee.json" - }, - { - "match": "or triggers list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "d1082fb9ed.json" - }, - { - "match": "or triggers list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "d1082fb9ed.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/uip-fixture.json new file mode 100644 index 0000000000..30a4a10e62 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/data/uip-fixture.json @@ -0,0 +1,120 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--process-name", + "ExcelDailyImport", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"ff666666-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:30:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:30:06.700Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"ff777777-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:00:05.900Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"ff666666-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:30:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:30:06.700Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n },\r\n {\r\n \"Key\": \"ff777777-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:00:05.900Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "triggers", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TriggersList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"t1111111-aaaa-bbbb-cccc-ddddeeeeffff\",\r\n \"Name\": \"HourlySalesA\",\r\n \"Enabled\": true,\r\n \"ProcessKey\": \"5680101\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"Cron\": \"0 0/30 * * * ?\",\r\n \"CronSummary\": \"Every 30 minutes\",\r\n \"TimeZoneId\": \"UTC\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"FolderName\": \"ExcelImports\",\r\n \"StartProcessCron\": \"0 0/30 * * * ?\",\r\n \"StartProcessCronSummary\": \"Every 30 minutes\",\r\n \"StartProcessCronDetails\": \"Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z\",\r\n \"JobsCount\": 1,\r\n \"RuntimeType\": \"Unattended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ee555555-6666-7777-8888-99990000aaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ee55555-6666-7777-8888-99990000aaaa\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\\r\\n ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\\r\\n at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)\\r\\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\\r\\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"ApplicationException\",\r\n \"ActivityName\": \"UseExcelFile_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"Id\": 680033001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ee555555-6666-7777-8888-99990000aaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ee555555-6666-7777-8888-99990000aaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ee555555-001\",\r\n \"JobKey\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee555555-002\",\r\n \"JobKey\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee555555-003\",\r\n \"JobKey\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.512Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee555555-004\",\r\n \"JobKey\": \"ee555555-6666-7777-8888-99990000aaaa\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.802Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/task.yaml index da195e5299..c449fe625f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-host-evidence/task.yaml @@ -25,21 +25,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/README.md new file mode 100644 index 0000000000..d1280b763a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/README.md @@ -0,0 +1,51 @@ +# Excel Read Range — Sensitivity Label Blocks Robot Access + +This scenario reproduces a Read Range failure where the workbook is +protected by a Microsoft Purview / Azure Information Protection +sensitivity label that the Robot user lacks permission for. The job +ends with: + +``` +System.NullReferenceException: Object reference not set to an instance of an object. +``` + +(no cell pointer, no sheet name). + +## What this scenario uncovers + +**Root Cause:** The workbook `C:\Robot\Data\sales-2026-05.xlsx` is +labeled `Confidential\Limited Access`. The interactive workbook +owner has the label permission and can read the file from Explorer. +The Robot user (`UIPATH\AUTOMATION1`) is not in the label's policy, +so when the `Use Excel File` activity opens the file under the +Robot's identity, the label's encryption / policy engine refuses +programmatic access. The activity's parser receives a null where it +expected workbook content and throws `NullReferenceException`. + +The user's "Confidential banner at the top" clue + the job-log +Trace entry naming the label together pinpoint branch 1 of the +playbook. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` +(the "Sensitivity label" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` with absolute literal `WorkbookPath`, then `Read Range` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (bare NRE) → `jobs logs` +(Trace entry surfaces the label name) → workflow source → conclude +branch 1. + +> **Note on fixtures.** Synthetic. The label name `Confidential\Limited +> Access` and Robot user are placeholders representative of a real +> Purview deployment. The test grades whether the agent surfaces +> the Purview / AIP branch using the banner clue + the log evidence. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/480438252b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/480438252b.json deleted file mode 100644 index d0489eb21c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/480438252b.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/904f769e4a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/904f769e4a.json deleted file mode 100644 index d8da257caf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/904f769e4a.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa1111bb-001", - "JobKey": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111bb-002", - "JobKey": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111bb-003", - "JobKey": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — workbook opened (sensitivity label: 'Confidential\\Limited Access'; encryption: enabled; access principal: UIPATH\\AUTOMATION1; decision: AccessDenied)", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111bb-004", - "JobKey": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.412Z", - "Message": "Read Range: Data — executing (SheetName='Data')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa1111bb-005", - "JobKey": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Use Excel File: sales-2026-05.xlsx — System.NullReferenceException: Object reference not set to an instance of an object.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/README.md deleted file mode 100644 index 1b96812a75..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/README.md +++ /dev/null @@ -1,50 +0,0 @@ -# Excel Read Range — Sensitivity Label Blocks Robot Access - -This scenario reproduces a Read Range failure where the workbook is -protected by a Microsoft Purview / Azure Information Protection -sensitivity label that the Robot user lacks permission for. The job -ends with: - -``` -System.NullReferenceException: Object reference not set to an instance of an object. -``` - -(no cell pointer, no sheet name). - -## What this scenario uncovers - -**Root Cause:** The workbook `C:\Robot\Data\sales-2026-05.xlsx` is -labeled `Confidential\Limited Access`. The interactive workbook -owner has the label permission and can read the file from Explorer. -The Robot user (`UIPATH\AUTOMATION1`) is not in the label's policy, -so when the `Use Excel File` activity opens the file under the -Robot's identity, the label's encryption / policy engine refuses -programmatic access. The activity's parser receives a null where it -expected workbook content and throws `NullReferenceException`. - -The user's "Confidential banner at the top" clue + the job-log -Trace entry naming the label together pinpoint branch 1 of the -playbook. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` -(the "Sensitivity label" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` with absolute literal `WorkbookPath`, then `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entry from `Use Excel File` includes the label name (`Confidential\Limited Access`) | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (bare NRE) → `jobs logs` -(Trace entry surfaces the label name) → workflow source → conclude -branch 1. - -> **Note on fixtures.** Synthetic. The label name `Confidential\Limited -> Access` and Robot user are placeholders representative of a real -> Purview deployment. The test grades whether the agent surfaces -> the Purview / AIP branch using the banner clue + the log evidence. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/c9999c412c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/c9999c412c.json deleted file mode 100644 index e14aa5410d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/c9999c412c.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/f5751f65d1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/f5751f65d1.json deleted file mode 100644 index 90c2d5fcaf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/f5751f65d1.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-bbbb-cccc-dddd-eeeeffffaaaa", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "NullReferenceException", - "ActivityName": "UseExcelFile_1", - "ActivityDisplayName": "Use Excel File: sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", - "Id": 680088001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/manifest.json deleted file mode 100644 index 55d14d62ba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "480438252b.json"}, - {"match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", "file": "480438252b.json"}, - {"match": "or jobs get \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "f5751f65d1.json"}, - {"match": "or jobs get aa111111-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "f5751f65d1.json"}, - {"match": "or jobs history \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "c9999c412c.json"}, - {"match": "or jobs history aa111111-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "c9999c412c.json"}, - {"match": "or jobs logs \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "904f769e4a.json"}, - {"match": "or jobs logs aa111111-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "904f769e4a.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/uip-fixture.json new file mode 100644 index 0000000000..e5f3e09701 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"NullReferenceException\",\r\n \"ActivityName\": \"UseExcelFile_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Id\": 680088001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa111111-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa1111bb-001\",\r\n \"JobKey\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111bb-002\",\r\n \"JobKey\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111bb-003\",\r\n \"JobKey\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — workbook opened (sensitivity label: 'Confidential\\\\Limited Access'; encryption: enabled; access principal: UIPATH\\\\AUTOMATION1; decision: AccessDenied)\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111bb-004\",\r\n \"JobKey\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.412Z\",\r\n \"Message\": \"Read Range: Data — executing (SheetName='Data')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa1111bb-005\",\r\n \"JobKey\": \"aa111111-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/task.yaml index 11128db2d3..dea593a51c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-label/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/README.md new file mode 100644 index 0000000000..fd2be36260 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/README.md @@ -0,0 +1,48 @@ +# Excel Read Range — Broken Named Range (#REF!) + +This scenario reproduces a Read Range failure where the workflow +references a named range (`Range: "MyDataRange"`), the workbook +opens successfully, sheet enumeration succeeds, but the named-range +lookup fails because the named range's `Refers to` in Name Manager +is `#REF!`. The job ends with: + +``` +System.NullReferenceException: Object reference not set to an instance of an object. +``` + +(no cell pointer). + +## What this scenario uncovers + +**Root Cause:** The workbook's defined name `MyDataRange` previously +pointed at `Sheet1!A1:E1000`, but `Sheet1` was deleted or restructured +upstream. The named range's reference is now `#REF!`. The workflow's +Read Range with `Range: "MyDataRange"` looks up the name in the +workbook's defined-names collection, gets a broken-reference target, +and throws `NullReferenceException` from inside the activity's +range-resolution code. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` +(the "Broken named range / formula reference" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` → `Get Workbook Sheets` (succeeds) → `Read Range "MyDataRange"` (NRE) | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (bare NRE) → `jobs logs` +(Get Workbook Sheets succeeded; Read Range Trace shows named-range +lookup with `RefersTo: #REF!`) → workflow source (named-range Range +property) → conclude branch 3. + +> **Note on fixtures.** Synthetic. The named range `MyDataRange` +> and the broken refers-to are placeholders. The test grades +> whether the agent surfaces the named-range branch using the +> workflow-source + log evidence. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/4ae039d222.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/4ae039d222.json deleted file mode 100644 index 4bc1a8e515..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/4ae039d222.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/README.md deleted file mode 100644 index 4a0e361c46..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Excel Read Range — Broken Named Range (#REF!) - -This scenario reproduces a Read Range failure where the workflow -references a named range (`Range: "MyDataRange"`), the workbook -opens successfully, sheet enumeration succeeds, but the named-range -lookup fails because the named range's `Refers to` in Name Manager -is `#REF!`. The job ends with: - -``` -System.NullReferenceException: Object reference not set to an instance of an object. -``` - -(no cell pointer). - -## What this scenario uncovers - -**Root Cause:** The workbook's defined name `MyDataRange` previously -pointed at `Sheet1!A1:E1000`, but `Sheet1` was deleted or restructured -upstream. The named range's reference is now `#REF!`. The workflow's -Read Range with `Range: "MyDataRange"` looks up the name in the -workbook's defined-names collection, gets a broken-reference target, -and throws `NullReferenceException` from inside the activity's -range-resolution code. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` -(the "Broken named range / formula reference" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` → `Get Workbook Sheets` (succeeds) → `Read Range "MyDataRange"` (NRE) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entry from `Read Range` includes the named-range lookup attempt with the broken refers-to value | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (bare NRE) → `jobs logs` -(Get Workbook Sheets succeeded; Read Range Trace shows named-range -lookup with `RefersTo: #REF!`) → workflow source (named-range Range -property) → conclude branch 3. - -> **Note on fixtures.** Synthetic. The named range `MyDataRange` -> and the broken refers-to are placeholders. The test grades -> whether the agent surfaces the named-range branch using the -> workflow-source + log evidence. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/cf6cba2cb7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/cf6cba2cb7.json deleted file mode 100644 index b6e5800cb5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/cf6cba2cb7.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/e5c94d47b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/e5c94d47b6.json deleted file mode 100644 index e8809267bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/e5c94d47b6.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-bbbb-cccc-dddd-eeeeffffaaaa", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveNamedRange(IExcelWorkbook workbook, String namedRange)\r\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "NullReferenceException", - "ActivityName": "ExcelReadRangeX_1", - "ActivityDisplayName": "Read Range: MyDataRange", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Id": 680088002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/f136ac9fce.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/f136ac9fce.json deleted file mode 100644 index d3f9572805..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/f136ac9fce.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-bb2222bb-001", - "JobKey": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222bb-002", - "JobKey": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading named range MyDataRange from C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222bb-003", - "JobKey": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — workbook opened", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222bb-004", - "JobKey": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.812Z", - "Message": "[ExcelDailyImport] Available sheets in workbook: [\"Data\", \"Summary\", \"Archive\"]", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222bb-005", - "JobKey": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.412Z", - "Message": "Read Range: MyDataRange — resolving named range 'MyDataRange' (RefersTo: '#REF!'; resolved sheet: ; resolved address: )", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb2222bb-006", - "JobKey": "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Read Range: MyDataRange — System.NullReferenceException: Object reference not set to an instance of an object.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/manifest.json deleted file mode 100644 index ab6239f22a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "cf6cba2cb7.json"}, - {"match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", "file": "cf6cba2cb7.json"}, - {"match": "or jobs get \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "e5c94d47b6.json"}, - {"match": "or jobs get bb222222-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "e5c94d47b6.json"}, - {"match": "or jobs history \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "4ae039d222.json"}, - {"match": "or jobs history bb222222-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "4ae039d222.json"}, - {"match": "or jobs logs \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "f136ac9fce.json"}, - {"match": "or jobs logs bb222222-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "f136ac9fce.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/uip-fixture.json new file mode 100644 index 0000000000..fe448214d0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveNamedRange(IExcelWorkbook workbook, String namedRange)\\r\\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"NullReferenceException\",\r\n \"ActivityName\": \"ExcelReadRangeX_1\",\r\n \"ActivityDisplayName\": \"Read Range: MyDataRange\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Id\": 680088002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb222222-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-bb2222bb-001\",\r\n \"JobKey\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222bb-002\",\r\n \"JobKey\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading named range MyDataRange from C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222bb-003\",\r\n \"JobKey\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — workbook opened\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222bb-004\",\r\n \"JobKey\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.812Z\",\r\n \"Message\": \"[ExcelDailyImport] Available sheets in workbook: [\\\"Data\\\", \\\"Summary\\\", \\\"Archive\\\"]\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222bb-005\",\r\n \"JobKey\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.412Z\",\r\n \"Message\": \"Read Range: MyDataRange — resolving named range 'MyDataRange' (RefersTo: '#REF!'; resolved sheet: ; resolved address: )\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb2222bb-006\",\r\n \"JobKey\": \"bb222222-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Read Range: MyDataRange — System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/task.yaml index bf02dc8b7e..b13cca93b0 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-namedrange/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/README.md new file mode 100644 index 0000000000..6f6c285b3b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/README.md @@ -0,0 +1,62 @@ +# Excel Read Range — Opaque NRE / TIE, Needs Workbook Inspection + +This scenario reproduces a Read Range failure where the workbook +opens successfully but parsing fails with a generic +`TargetInvocationException` wrapping an inner `NullReferenceException`. +The CLI evidence rules out the easier branches but cannot positively +identify which of the remaining (corruption / OpenXML feature / scale) +applies — diagnosis requires manual workbook inspection. The job +ends with: + +``` +System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. + ---> System.NullReferenceException: Object reference not set to an instance of an object. +``` + +## What this scenario uncovers + +**Expected outcome:** The agent matches the playbook, recognizes +that: +- No sensitivity-label evidence in job logs (rules out branch 1). +- Workflow source uses an empty `Range` (not a named range), ruling + out branch 3. +- The remaining candidates (branch 2 corruption / branch 4 + unsupported OpenXML feature / branch 5 heavy formatting) cannot + be distinguished by CLI alone. + +The agent then hands the user a workbook-inspection checklist (open +in Excel; look for "Repaired" dialog; Name Manager `Ctrl+F3` for +`#REF!`; Compatibility Checker; file size). + +**Bonus signal:** the user mentions the workbook is "generated by +a Python script that runs nightly." This is a strong but not +conclusive lean toward branch 2 (structural corruption — `openpyxl` +and similar libraries frequently produce structurally invalid +XLSX). The agent may name this as the most-likely candidate but +should still recommend verification before committing. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` +(collapsed branches 2/4/5 — opaque NRE / TIE). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` with empty `Range`, no named-range, no COM-forcing properties | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (TIE / NRE, no pointer) → +`jobs logs` (workbook opened OK; parser failed; no specific +branch-2/4/5 hint) → workflow source review (clean) → STOP. At +this point CLI evidence is exhausted; agent hands the user the +manual-inspection checklist. + +> **Note on fixtures.** Synthetic. Job key, folder key, and the +> Python-script clue are placeholders. The test grades whether +> the agent surfaces the manual-inspection checklist rather than +> confidently guessing a specific branch. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/2d49c3ab3c.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/2d49c3ab3c.json deleted file mode 100644 index 926b762174..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/2d49c3ab3c.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc3333bb-001", - "JobKey": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333bb-002", - "JobKey": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx (generated nightly by upstream Python script)", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333bb-003", - "JobKey": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — workbook opened (no sensitivity label detected; encryption: not enabled)", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333bb-004", - "JobKey": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.412Z", - "Message": "Read Range: Sheet1 (used range) — executing (SheetName='Sheet1'; Range='' [used range])", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc3333bb-005", - "JobKey": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Read Range: Sheet1 (used range) — System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/9a56a36c36.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/9a56a36c36.json deleted file mode 100644 index 92701c37fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/9a56a36c36.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/README.md deleted file mode 100644 index 918827ec80..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/README.md +++ /dev/null @@ -1,61 +0,0 @@ -# Excel Read Range — Opaque NRE / TIE, Needs Workbook Inspection - -This scenario reproduces a Read Range failure where the workbook -opens successfully but parsing fails with a generic -`TargetInvocationException` wrapping an inner `NullReferenceException`. -The CLI evidence rules out the easier branches but cannot positively -identify which of the remaining (corruption / OpenXML feature / scale) -applies — diagnosis requires manual workbook inspection. The job -ends with: - -``` -System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. - ---> System.NullReferenceException: Object reference not set to an instance of an object. -``` - -## What this scenario uncovers - -**Expected outcome:** The agent matches the playbook, recognizes -that: -- No sensitivity-label evidence in job logs (rules out branch 1). -- Workflow source uses an empty `Range` (not a named range), ruling - out branch 3. -- The remaining candidates (branch 2 corruption / branch 4 - unsupported OpenXML feature / branch 5 heavy formatting) cannot - be distinguished by CLI alone. - -The agent then hands the user a workbook-inspection checklist (open -in Excel; look for "Repaired" dialog; Name Manager `Ctrl+F3` for -`#REF!`; Compatibility Checker; file size). - -**Bonus signal:** the user mentions the workbook is "generated by -a Python script that runs nightly." This is a strong but not -conclusive lean toward branch 2 (structural corruption — `openpyxl` -and similar libraries frequently produce structurally invalid -XLSX). The agent may name this as the most-likely candidate but -should still recommend verification before committing. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/null-reference-from-activity.md` -(collapsed branches 2/4/5 — opaque NRE / TIE). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` with empty `Range`, no named-range, no COM-forcing properties | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field shows TIE wrapping NRE; the `or jobs logs` shows no specific cell pointer and no label / named-range evidence | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (TIE / NRE, no pointer) → -`jobs logs` (workbook opened OK; parser failed; no specific -branch-2/4/5 hint) → workflow source review (clean) → STOP. At -this point CLI evidence is exhausted; agent hands the user the -manual-inspection checklist. - -> **Note on fixtures.** Synthetic. Job key, folder key, and the -> Python-script clue are placeholders. The test grades whether -> the agent surfaces the manual-inspection checklist rather than -> confidently guessing a specific branch. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/cbca3820c4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/cbca3820c4.json deleted file mode 100644 index 8e4850d7d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/cbca3820c4.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9cc33333-bbbb-cccc-dddd-eeeeffffaaaa", - "Info": "System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.\r\n ---> System.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\r\n at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)\r\n --- End of inner exception stack trace ---\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "TargetInvocationException", - "ActivityName": "ExcelReadRangeX_1", - "ActivityDisplayName": "Read Range: Sheet1 (used range)", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "Id": 680088003 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/f5469e8bd7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/f5469e8bd7.json deleted file mode 100644 index 3311d847ee..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/f5469e8bd7.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/manifest.json deleted file mode 100644 index 74c18d393e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or folders list --output json", "file": "858738168e.json"}, - {"match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "f5469e8bd7.json"}, - {"match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", "file": "f5469e8bd7.json"}, - {"match": "or jobs get \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "cbca3820c4.json"}, - {"match": "or jobs get cc333333-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "cbca3820c4.json"}, - {"match": "or jobs history \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "9a56a36c36.json"}, - {"match": "or jobs history cc333333-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "9a56a36c36.json"}, - {"match": "or jobs logs \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\" --output json", "file": "2d49c3ab3c.json"}, - {"match": "or jobs logs cc333333-bbbb-cccc-dddd-eeeeffffaaaa --output json", "file": "2d49c3ab3c.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/uip-fixture.json new file mode 100644 index 0000000000..8b314a6b47 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9cc33333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Info\": \"System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.\\r\\n ---> System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\\r\\n at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)\\r\\n --- End of inner exception stack trace ---\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"TargetInvocationException\",\r\n \"ActivityName\": \"ExcelReadRangeX_1\",\r\n \"ActivityDisplayName\": \"Read Range: Sheet1 (used range)\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Id\": 680088003\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cc333333-bbbb-cccc-dddd-eeeeffffaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc3333bb-001\",\r\n \"JobKey\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333bb-002\",\r\n \"JobKey\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx (generated nightly by upstream Python script)\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333bb-003\",\r\n \"JobKey\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — workbook opened (no sensitivity label detected; encryption: not enabled)\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333bb-004\",\r\n \"JobKey\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.412Z\",\r\n \"Message\": \"Read Range: Sheet1 (used range) — executing (SheetName='Sheet1'; Range='' [used range])\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc3333bb-005\",\r\n \"JobKey\": \"cc333333-bbbb-cccc-dddd-eeeeffffaaaa\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Read Range: Sheet1 (used range) — System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/task.yaml index e4cc8bf4a1..8b9bc3a2ea 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-nre-opaque/task.yaml @@ -28,21 +28,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/README.md new file mode 100644 index 0000000000..d3f5883305 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/README.md @@ -0,0 +1,58 @@ +# Excel Read Range — File Locked by Orphan EXCEL.EXE From Prior Job + +This scenario reproduces an `ExcelReadRange` failure caused by **an +orphan EXCEL.EXE process left behind by the previous run of the same +process**. The job ends with: + +``` +System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. +``` + +## What this scenario uncovers + +**Root Cause:** The prior run of `ExcelDailyImport` (key +`dd444444-...`) ran 10 minutes earlier and faulted with a workflow +exception **inside** the `Excel Application Scope`. Because the scope +was authored with `ContinueOnError="True"` AND wrapped in a `Try Catch` +whose `Catch` swallowed the exception without disposing the scope, the +COM-launched `EXCEL.EXE` process was left running under the Robot +user's session. It still holds the workbook's file lock when the +current job (key `cc333333-...`) tries to open the same workbook. + +This is NOT a concurrent-jobs failure — there is only one trigger and +no sibling job is running. The CLI evidence chain is: + +- `or jobs list` shows the prior faulted run + the current run, no + overlap. +- Workflow source (`process/Main.xaml`) has + `ContinueOnError="True"` on the Excel Application Scope and a Try + Catch that does not re-throw. + +Together they pinpoint the orphan-process branch. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/workbook-file-locked.md` +(the "Orphan EXCEL.EXE from a prior job" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project where `Main.xaml` deliberately has `ContinueOnError="True"` on `ExcelApplicationScope` AND a `TryCatch` whose `Catch` only logs without re-throwing | + +The expected investigation chain: `folders list-current-user` → `jobs +list --state Faulted` → `jobs get` (current) → `jobs logs` → `jobs +list` (broader, surfaces the prior faulted run) → `jobs get ` +→ `triggers list` (single trigger, no overlap) → workflow source +inspection. + +> **Note on fixtures.** Like the sibling scenarios, fixtures are +> synthetic. The job keys, folder key, workbook path, and trigger +> name are placeholders — the test grades whether the agent surfaces +> the orphan-process branch with BOTH pieces of evidence (prior +> faulted job + `ContinueOnError="True"` in workflow source), not +> the specific names. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/30c10560f1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/30c10560f1.json deleted file mode 100644 index 793a366c78..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/30c10560f1.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dd444444-5555-6666-7777-888899990000", - "StartTime": "2026-05-19T07:50:00.500Z", - "EndTime": "2026-05-19T07:50:04.200Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9dd44444-5555-6666-7777-888899990000", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Excel.Activities.ExcelReadRange.Execute(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\nCaught and swallowed by TryCatch_1 in Main.xaml (Catch logs but does not re-throw).", - "CreationTime": "2026-05-19T07:50:00.205Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T07:50:04.200Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "NullReferenceException", - "ActivityName": "ExcelReadRange_1", - "ActivityDisplayName": "Read Range: Sheet1!A1", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "dd444444-5555-6666-7777-888899990000", - "Id": 680022002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/6c8b4e8b2a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/6c8b4e8b2a.json deleted file mode 100644 index 088514d840..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/6c8b4e8b2a.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-dd444444-001", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Info", - "Timestamp": "2026-05-19T07:50:00.510Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444444-002", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Info", - "Timestamp": "2026-05-19T07:50:00.620Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444444-003", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Trace", - "Timestamp": "2026-05-19T07:50:00.812Z", - "Message": "Excel Application Scope: sales-2026-05.xlsx — executing (Visible=True, AutoSave=True, ContinueOnError=True)", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444444-004", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Trace", - "Timestamp": "2026-05-19T07:50:03.118Z", - "Message": "Read Range: Sheet1!A1 — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444444-005", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Error", - "Timestamp": "2026-05-19T07:50:03.405Z", - "Message": "Read Range: Sheet1!A1 — System.NullReferenceException: Object reference not set to an instance of an object.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444444-006", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Warn", - "Timestamp": "2026-05-19T07:50:03.512Z", - "Message": "[ExcelDailyImport] Caught System.NullReferenceException: Object reference not set to an instance of an object — continuing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444444-007", - "JobKey": "dd444444-5555-6666-7777-888899990000", - "Level": "Info", - "Timestamp": "2026-05-19T07:50:04.118Z", - "Message": "[ExcelDailyImport] Import step finished (errors, if any, were logged).", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/README.md deleted file mode 100644 index 0359753b1f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# Excel Read Range — File Locked by Orphan EXCEL.EXE From Prior Job - -This scenario reproduces an `ExcelReadRange` failure caused by **an -orphan EXCEL.EXE process left behind by the previous run of the same -process**. The job ends with: - -``` -System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. -``` - -## What this scenario uncovers - -**Root Cause:** The prior run of `ExcelDailyImport` (key -`dd444444-...`) ran 10 minutes earlier and faulted with a workflow -exception **inside** the `Excel Application Scope`. Because the scope -was authored with `ContinueOnError="True"` AND wrapped in a `Try Catch` -whose `Catch` swallowed the exception without disposing the scope, the -COM-launched `EXCEL.EXE` process was left running under the Robot -user's session. It still holds the workbook's file lock when the -current job (key `cc333333-...`) tries to open the same workbook. - -This is NOT a concurrent-jobs failure — there is only one trigger and -no sibling job is running. The CLI evidence chain is: - -- `or jobs list` shows the prior faulted run + the current run, no - overlap. -- Workflow source (`process/Main.xaml`) has - `ContinueOnError="True"` on the Excel Application Scope and a Try - Catch that does not re-throw. - -Together they pinpoint the orphan-process branch. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/workbook-file-locked.md` -(the "Orphan EXCEL.EXE from a prior job" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | `ExcelDailyImport` project where `Main.xaml` deliberately has `ContinueOnError="True"` on `ExcelApplicationScope` AND a `TryCatch` whose `Catch` only logs without re-throwing | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The expected investigation chain: `folders list-current-user` → `jobs -list --state Faulted` → `jobs get` (current) → `jobs logs` → `jobs -list` (broader, surfaces the prior faulted run) → `jobs get ` -→ `triggers list` (single trigger, no overlap) → workflow source -inspection. - -> **Note on fixtures.** Like the sibling scenarios, fixtures are -> synthetic. The job keys, folder key, workbook path, and trigger -> name are placeholders — the test grades whether the agent surfaces -> the orphan-process branch with BOTH pieces of evidence (prior -> faulted job + `ContinueOnError="True"` in workflow source), not -> the specific names. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/cec128c294.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/cec128c294.json deleted file mode 100644 index ae983f9405..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/cec128c294.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-4444-5555-6666-777788889999", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - }, - { - "Key": "dd444444-5555-6666-7777-888899990000", - "StartTime": "2026-05-19T07:50:00.500Z", - "EndTime": "2026-05-19T07:50:04.200Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object. (Caught and swallowed by TryCatch_1 in Main.xaml.)", - "ErrorCode": "Activity" - }, - { - "Key": "ee555555-1111-2222-3333-444455556666", - "StartTime": "2026-05-19T07:20:00.500Z", - "EndTime": "2026-05-19T07:20:06.700Z", - "State": "Successful", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": null, - "ErrorCode": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d0ed234d13.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d0ed234d13.json deleted file mode 100644 index 1ed9f16831..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d0ed234d13.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-4444-5555-6666-777788889999", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9cc33333-4444-5555-6666-777788889999", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'.\r\n ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.\r\n at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)\r\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecuteAsync(NativeActivityContext context)\r\n --- End of inner exception stack trace ---\r\n", - "CreationTime": "2026-05-19T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:01.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "ApplicationException", - "ActivityName": "ExcelApplicationScope_1", - "ActivityDisplayName": "Excel Application Scope: sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-4444-5555-6666-777788889999", - "Id": 680022001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d1082fb9ed.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d1082fb9ed.json deleted file mode 100644 index f77898b138..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d1082fb9ed.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "TriggersList", - "Data": [ - { - "Key": "t1111111-aaaa-bbbb-cccc-ddddeeeeffff", - "Name": "HourlySalesA", - "Enabled": true, - "ProcessKey": "5680101", - "ProcessName": "ExcelDailyImport", - "Cron": "0 0/30 * * * ?", - "CronSummary": "Every 30 minutes", - "TimeZoneId": "UTC", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "FolderName": "ExcelImports", - "StartProcessCron": "0 0/30 * * * ?", - "StartProcessCronSummary": "Every 30 minutes", - "StartProcessCronDetails": "Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z", - "JobsCount": 1, - "RuntimeType": "Unattended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d13289c93f.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d13289c93f.json deleted file mode 100644 index 7bb2abba77..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d13289c93f.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "cc333333-4444-5555-6666-777788889999", - "State": "Pending", - "Timestamp": "2026-05-19T08:00:01.005Z" - }, - { - "Key": "cc333333-4444-5555-6666-777788889999", - "State": "Running", - "Timestamp": "2026-05-19T08:00:01.300Z" - }, - { - "Key": "cc333333-4444-5555-6666-777788889999", - "State": "Faulted", - "Timestamp": "2026-05-19T08:00:01.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d9bd348e9d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d9bd348e9d.json deleted file mode 100644 index 8fcc626907..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/d9bd348e9d.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-4444-5555-6666-777788889999", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - }, - { - "Key": "dd444444-5555-6666-7777-888899990000", - "StartTime": "2026-05-19T07:50:00.500Z", - "EndTime": "2026-05-19T07:50:04.200Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object. (Caught and swallowed by TryCatch_1 in Main.xaml.)", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/dde283e718.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/dde283e718.json deleted file mode 100644 index eb87759285..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/dde283e718.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc333333-001", - "JobKey": "cc333333-4444-5555-6666-777788889999", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333333-002", - "JobKey": "cc333333-4444-5555-6666-777788889999", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333333-003", - "JobKey": "cc333333-4444-5555-6666-777788889999", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.512Z", - "Message": "Excel Application Scope: sales-2026-05.xlsx — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333333-004", - "JobKey": "cc333333-4444-5555-6666-777788889999", - "Level": "Error", - "Timestamp": "2026-05-19T08:00:01.802Z", - "Message": "Excel Application Scope: sales-2026-05.xlsx — UiPath.Excel.ExcelException: Error opening workbook 'C:\\Robot\\Data\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/manifest.json deleted file mode 100644 index c717b25846..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/m/r/manifest.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "d9bd348e9d.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "d9bd348e9d.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "cec128c294.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "cec128c294.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --process-name \"ExcelDailyImport\" --output json", - "file": "cec128c294.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --process-name ExcelDailyImport --output json", - "file": "cec128c294.json" - }, - { - "match": "or jobs get \"cc333333-4444-5555-6666-777788889999\" --output json", - "file": "d0ed234d13.json" - }, - { - "match": "or jobs get cc333333-4444-5555-6666-777788889999 --output json", - "file": "d0ed234d13.json" - }, - { - "match": "or jobs get \"dd444444-5555-6666-7777-888899990000\" --output json", - "file": "30c10560f1.json" - }, - { - "match": "or jobs get dd444444-5555-6666-7777-888899990000 --output json", - "file": "30c10560f1.json" - }, - { - "match": "or jobs history \"cc333333-4444-5555-6666-777788889999\" --output json", - "file": "d13289c93f.json" - }, - { - "match": "or jobs history cc333333-4444-5555-6666-777788889999 --output json", - "file": "d13289c93f.json" - }, - { - "match": "or jobs logs \"cc333333-4444-5555-6666-777788889999\" --output json", - "file": "dde283e718.json" - }, - { - "match": "or jobs logs cc333333-4444-5555-6666-777788889999 --output json", - "file": "dde283e718.json" - }, - { - "match": "or jobs logs \"dd444444-5555-6666-7777-888899990000\" --output json", - "file": "6c8b4e8b2a.json" - }, - { - "match": "or jobs logs dd444444-5555-6666-7777-888899990000 --output json", - "file": "6c8b4e8b2a.json" - }, - { - "match": "or triggers list --folder-key \"f0011111-2222-3333-4444-555566667777\" --output json", - "file": "d1082fb9ed.json" - }, - { - "match": "or triggers list --folder-key f0011111-2222-3333-4444-555566667777 --output json", - "file": "d1082fb9ed.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/uip-fixture.json new file mode 100644 index 0000000000..e43614c9ee --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/data/uip-fixture.json @@ -0,0 +1,146 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"StartTime\": \"2026-05-19T07:50:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:50:04.200Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object. (Caught and swallowed by TryCatch_1 in Main.xaml.)\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--process-name", + "ExcelDailyImport", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"StartTime\": \"2026-05-19T07:50:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:50:04.200Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object. (Caught and swallowed by TryCatch_1 in Main.xaml.)\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"ee555555-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:20:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:20:06.700Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"StartTime\": \"2026-05-19T07:50:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:50:04.200Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object. (Caught and swallowed by TryCatch_1 in Main.xaml.)\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"ee555555-1111-2222-3333-444455556666\",\r\n \"StartTime\": \"2026-05-19T07:20:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:20:06.700Z\",\r\n \"State\": \"Successful\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": null,\r\n \"ErrorCode\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "triggers", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TriggersList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"t1111111-aaaa-bbbb-cccc-ddddeeeeffff\",\r\n \"Name\": \"HourlySalesA\",\r\n \"Enabled\": true,\r\n \"ProcessKey\": \"5680101\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"Cron\": \"0 0/30 * * * ?\",\r\n \"CronSummary\": \"Every 30 minutes\",\r\n \"TimeZoneId\": \"UTC\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"FolderName\": \"ExcelImports\",\r\n \"StartProcessCron\": \"0 0/30 * * * ?\",\r\n \"StartProcessCronSummary\": \"Every 30 minutes\",\r\n \"StartProcessCronDetails\": \"Minutes: 0,30; Seconds: 0; Anchor: 2026-05-01T00:00:00Z\",\r\n \"JobsCount\": 1,\r\n \"RuntimeType\": \"Unattended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cc333333-4444-5555-6666-777788889999", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9cc33333-4444-5555-6666-777788889999\",\r\n \"Info\": \"UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'.\\r\\n ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\\r\\n at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)\\r\\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecuteAsync(NativeActivityContext context)\\r\\n --- End of inner exception stack trace ---\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:01.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"ApplicationException\",\r\n \"ActivityName\": \"ExcelApplicationScope_1\",\r\n \"ActivityDisplayName\": \"Excel Application Scope: sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"Id\": 680022001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "dd444444-5555-6666-7777-888899990000", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"StartTime\": \"2026-05-19T07:50:00.500Z\",\r\n \"EndTime\": \"2026-05-19T07:50:04.200Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9dd44444-5555-6666-7777-888899990000\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Excel.Activities.ExcelReadRange.Execute(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\nCaught and swallowed by TryCatch_1 in Main.xaml (Catch logs but does not re-throw).\",\r\n \"CreationTime\": \"2026-05-19T07:50:00.205Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T07:50:04.200Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"NullReferenceException\",\r\n \"ActivityName\": \"ExcelReadRange_1\",\r\n \"ActivityDisplayName\": \"Read Range: Sheet1!A1\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Id\": 680022002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cc333333-4444-5555-6666-777788889999", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cc333333-4444-5555-6666-777788889999", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc333333-001\",\r\n \"JobKey\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333333-002\",\r\n \"JobKey\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333333-003\",\r\n \"JobKey\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.512Z\",\r\n \"Message\": \"Excel Application Scope: sales-2026-05.xlsx — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333333-004\",\r\n \"JobKey\": \"cc333333-4444-5555-6666-777788889999\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.802Z\",\r\n \"Message\": \"Excel Application Scope: sales-2026-05.xlsx — UiPath.Excel.ExcelException: Error opening workbook 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx'. ---> System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dd444444-5555-6666-7777-888899990000", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-dd444444-001\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T07:50:00.510Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444444-002\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T07:50:00.620Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444444-003\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T07:50:00.812Z\",\r\n \"Message\": \"Excel Application Scope: sales-2026-05.xlsx — executing (Visible=True, AutoSave=True, ContinueOnError=True)\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444444-004\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T07:50:03.118Z\",\r\n \"Message\": \"Read Range: Sheet1!A1 — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444444-005\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T07:50:03.405Z\",\r\n \"Message\": \"Read Range: Sheet1!A1 — System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444444-006\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Warn\",\r\n \"Timestamp\": \"2026-05-19T07:50:03.512Z\",\r\n \"Message\": \"[ExcelDailyImport] Caught System.NullReferenceException: Object reference not set to an instance of an object — continuing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444444-007\",\r\n \"JobKey\": \"dd444444-5555-6666-7777-888899990000\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T07:50:04.118Z\",\r\n \"Message\": \"[ExcelDailyImport] Import step finished (errors, if any, were logged).\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/task.yaml index d8c7f4fba7..347548649c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-orphan-process/task.yaml @@ -24,21 +24,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/README.md new file mode 100644 index 0000000000..5b159ed606 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/README.md @@ -0,0 +1,46 @@ +# Excel Read Range — Relative WorkbookPath Resolved Against Wrong CWD + +This scenario reproduces a Read Range failure where the workflow uses +a relative `WorkbookPath` that resolves to the per-package +`%LocalAppData%` directory at unattended runtime, instead of the user's +intended data folder. The job ends with: + +``` +System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\automation1\AppData\Local\UiPath\Packages\ExcelDailyImport\1.0.0\Data\sales-2026-05.xlsx'. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's `WorkbookPath` is the relative literal +`"Data\sales-2026-05.xlsx"`. The .NET runtime resolves relative paths +against `Environment.CurrentDirectory`. Studio's CurrentDirectory is +the project folder (so the workflow "works on my machine"). The +Robot's unattended runtime CWD is +`%LocalAppData%\UiPath\Packages\\\`, which has no +`Data\sales-2026-05.xlsx`. The resolved path's prefix is the smoking +gun. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-file-not-found.md` +(the "Relative path resolved against wrong CWD" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` with **relative** `WorkbookPath: "Data\sales-2026-05.xlsx"`, then `Read Range` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (resolved path shows +per-package prefix) → `jobs logs` → workflow source review (relative +literal `WorkbookPath`) → conclude branch 3. + +> **Note on fixtures.** Synthetic. The job key, folder key, and +> resolved-path prefix are placeholders representative of a real +> Robot host layout. The test grades whether the agent surfaces the +> CWD divergence using both the workflow-source evidence AND the +> resolved-path prefix. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/159669d361.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/159669d361.json deleted file mode 100644 index 480ae1170a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/159669d361.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc333399-001", - "JobKey": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-002", - "JobKey": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading workbook at Data\\sales-2026-05.xlsx (relative to CWD)", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-003", - "JobKey": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: Data\\sales-2026-05.xlsx — opening workbook (WorkbookPath='Data\\sales-2026-05.xlsx', resolved against CWD='C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\ExcelDailyImport\\1.0.0\\')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-004", - "JobKey": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:01.802Z", - "Message": "Use Excel File: Data\\sales-2026-05.xlsx — System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\ExcelDailyImport\\1.0.0\\Data\\sales-2026-05.xlsx'.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/32c77da18d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/32c77da18d.json deleted file mode 100644 index 37ada98daa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/32c77da18d.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "State": "Pending", - "Timestamp": "2026-05-20T08:00:01.005Z" - }, - { - "Key": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "State": "Running", - "Timestamp": "2026-05-20T08:00:01.300Z" - }, - { - "Key": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "State": "Faulted", - "Timestamp": "2026-05-20T08:00:01.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/3dbf2901c1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/3dbf2901c1.json deleted file mode 100644 index b1721948b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/3dbf2901c1.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\ExcelDailyImport\\1.0.0\\Data\\sales-2026-05.xlsx'.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/60e1c2d756.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/60e1c2d756.json deleted file mode 100644 index 6ca6541028..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/60e1c2d756.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9cc33333-9999-aaaa-bbbb-ccccddddeeee", - "Info": "System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\ExcelDailyImport\\1.0.0\\Data\\sales-2026-05.xlsx'.\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, ...)\r\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\r\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:01.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "DirectoryNotFoundException", - "ActivityName": "UseExcelFile_1", - "ActivityDisplayName": "Use Excel File: Data\\sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-9999-aaaa-bbbb-ccccddddeeee", - "Id": 680077003 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/README.md deleted file mode 100644 index 4526c23599..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Excel Read Range — Relative WorkbookPath Resolved Against Wrong CWD - -This scenario reproduces a Read Range failure where the workflow uses -a relative `WorkbookPath` that resolves to the per-package -`%LocalAppData%` directory at unattended runtime, instead of the user's -intended data folder. The job ends with: - -``` -System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\automation1\AppData\Local\UiPath\Packages\ExcelDailyImport\1.0.0\Data\sales-2026-05.xlsx'. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's `WorkbookPath` is the relative literal -`"Data\sales-2026-05.xlsx"`. The .NET runtime resolves relative paths -against `Environment.CurrentDirectory`. Studio's CurrentDirectory is -the project folder (so the workflow "works on my machine"). The -Robot's unattended runtime CWD is -`%LocalAppData%\UiPath\Packages\\\`, which has no -`Data\sales-2026-05.xlsx`. The resolved path's prefix is the smoking -gun. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-file-not-found.md` -(the "Relative path resolved against wrong CWD" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` with **relative** `WorkbookPath: "Data\sales-2026-05.xlsx"`, then `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field echoes the resolved path with the `%LocalAppData%\UiPath\Packages\...` prefix | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (resolved path shows -per-package prefix) → `jobs logs` → workflow source review (relative -literal `WorkbookPath`) → conclude branch 3. - -> **Note on fixtures.** Synthetic. The job key, folder key, and -> resolved-path prefix are placeholders representative of a real -> Robot host layout. The test grades whether the agent surfaces the -> CWD divergence using both the workflow-source evidence AND the -> resolved-path prefix. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/manifest.json deleted file mode 100644 index ba2584d87f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/m/r/manifest.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range daily-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "3dbf2901c1.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "3dbf2901c1.json" - }, - { - "match": "or jobs get \"cc333333-9999-aaaa-bbbb-ccccddddeeee\" --output json", - "file": "60e1c2d756.json" - }, - { - "match": "or jobs get cc333333-9999-aaaa-bbbb-ccccddddeeee --output json", - "file": "60e1c2d756.json" - }, - { - "match": "or jobs history \"cc333333-9999-aaaa-bbbb-ccccddddeeee\" --output json", - "file": "32c77da18d.json" - }, - { - "match": "or jobs history cc333333-9999-aaaa-bbbb-ccccddddeeee --output json", - "file": "32c77da18d.json" - }, - { - "match": "or jobs logs \"cc333333-9999-aaaa-bbbb-ccccddddeeee\" --output json", - "file": "159669d361.json" - }, - { - "match": "or jobs logs cc333333-9999-aaaa-bbbb-ccccddddeeee --output json", - "file": "159669d361.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/uip-fixture.json new file mode 100644 index 0000000000..21da179244 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\ExcelDailyImport\\\\1.0.0\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cc333333-9999-aaaa-bbbb-ccccddddeeee", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9cc33333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Info\": \"System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\ExcelDailyImport\\\\1.0.0\\\\Data\\\\sales-2026-05.xlsx'.\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, ...)\\r\\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\\r\\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"DirectoryNotFoundException\",\r\n \"ActivityName\": \"UseExcelFile_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: Data\\\\sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Id\": 680077003\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cc333333-9999-aaaa-bbbb-ccccddddeeee", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cc333333-9999-aaaa-bbbb-ccccddddeeee", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc333399-001\",\r\n \"JobKey\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-002\",\r\n \"JobKey\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading workbook at Data\\\\sales-2026-05.xlsx (relative to CWD)\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-003\",\r\n \"JobKey\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: Data\\\\sales-2026-05.xlsx — opening workbook (WorkbookPath='Data\\\\sales-2026-05.xlsx', resolved against CWD='C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\ExcelDailyImport\\\\1.0.0\\\\')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-004\",\r\n \"JobKey\": \"cc333333-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.802Z\",\r\n \"Message\": \"Use Excel File: Data\\\\sales-2026-05.xlsx — System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\ExcelDailyImport\\\\1.0.0\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/task.yaml index 61a2ed7800..04153b1ff4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-relative-path/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/README.md new file mode 100644 index 0000000000..7771dab3ed --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/README.md @@ -0,0 +1,72 @@ +# Excel Read Range — Sheet Name Visually Identical But Byte-Different + +This scenario reproduces a Read Range failure where the configured +`SheetName` and the workbook's actual sheet name **look identical +in every UI surface** but differ at the byte level — non-breaking +space `U+00A0` vs regular space `U+0020`. The job ends with: + +``` +UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist. +``` + +## What this scenario uncovers + +**Expected outcome:** The agent matches the playbook AND either +(a) identifies the `U+00A0` vs `U+0020` mismatch from the byte +content of the `Get Workbook Sheets` log evidence — the JSON tool +output preserves the real code points, and an LLM agent reads +bytes, not pixels — or (b) narrows to the whitespace/look-alike +branches and recommends the host-side byte-compare (the PowerShell +snippet from the playbook) to identify the code point. The agent +must NOT assert a specific code point without byte-level evidence, +and must NOT pick a branch (typo, case, rename) the CLI evidence +rules out. + +**The branches that COULD apply** without further evidence: + +- Leading/trailing whitespace (branch 5) — visible only when + inspecting byte-by-byte. +- Look-alike Unicode character (branch 6) — NBSP, Cyrillic vs + Latin letters, curly vs straight quotes; visually identical in + every UI surface. + +**The branches that are ruled OUT by the CLI evidence:** + +- Typo (branch 1) — configured and logged names are visually + identical. +- Case mismatch (branch 2) — visually identical, same case. +- Sheet renamed (branch 3) — the configured name appears in the + Get Workbook Sheets log output. +- Sheet deleted (branch 4) — the configured name appears in the + Get Workbook Sheets log output. +- Dynamic expression resolved wrong (branch 7) — workflow source + shows a literal `SheetName: "Quarterly Data"`. + +The agent may resolve between the remaining branches by inspecting +the bytes the job-log evidence already carries; what it must NOT do +is assert a specific code point it never inspected. Unsupported +confident guessing is the failure mode this test catches. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` → `Get Workbook Sheets` (logs result) → `Read Range "Quarterly Data"` (fails). Configured name uses regular space `U+0020` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` → `jobs logs` → workflow +source review → recognize that all visible names match → identify +the NBSP from the log bytes, or recommend the host-side +byte-compare command from the playbook. + +> **Note on fixtures.** Synthetic. The actual sheet name in the +> fixture log message is stored as a real non-breaking space byte +> in the mock response. It renders identically to a regular space +> for a human reader, but the bytes reach the agent's tool output +> intact — so byte-level identification is a legitimate diagnosis +> path. Keep the raw NBSP byte in the mock (a six-character +> backslash-u escape would disclose the answer as plain ASCII). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/22546d1c6d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/22546d1c6d.json deleted file mode 100644 index b3b77e72b2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/22546d1c6d.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc333377-001", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333377-002", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333377-003", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333377-004", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.612Z", - "Message": "Get Workbook Sheets — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333377-005", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.812Z", - "Message": "[ExcelDailyImport] Available sheets in workbook: [\"Sheet1\", \"Quarterly Data\", \"Summary\"]", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333377-006", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:02.512Z", - "Message": "Read Range: Quarterly Data!A1 — executing (SheetName='Quarterly Data')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333377-007", - "JobKey": "cc333333-7777-8888-9999-000011112222", - "Level": "Error", - "Timestamp": "2026-05-19T08:00:02.802Z", - "Message": "Read Range: Quarterly Data!A1 — UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/61d7a8b016.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/61d7a8b016.json deleted file mode 100644 index 80121350a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/61d7a8b016.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "cc333333-7777-8888-9999-000011112222", - "State": "Pending", - "Timestamp": "2026-05-19T08:00:01.005Z" - }, - { - "Key": "cc333333-7777-8888-9999-000011112222", - "State": "Running", - "Timestamp": "2026-05-19T08:00:01.300Z" - }, - { - "Key": "cc333333-7777-8888-9999-000011112222", - "State": "Faulted", - "Timestamp": "2026-05-19T08:00:02.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/6b808e9e6a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/6b808e9e6a.json deleted file mode 100644 index e065d821a1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/6b808e9e6a.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-7777-8888-9999-000011112222", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/README.md deleted file mode 100644 index 22e8fae53a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/README.md +++ /dev/null @@ -1,71 +0,0 @@ -# Excel Read Range — Sheet Name Visually Identical But Byte-Different - -This scenario reproduces a Read Range failure where the configured -`SheetName` and the workbook's actual sheet name **look identical -in every UI surface** but differ at the byte level — non-breaking -space `U+00A0` vs regular space `U+0020`. The job ends with: - -``` -UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist. -``` - -## What this scenario uncovers - -**Expected outcome:** The agent matches the playbook AND either -(a) identifies the `U+00A0` vs `U+0020` mismatch from the byte -content of the `Get Workbook Sheets` log evidence — the JSON tool -output preserves the real code points, and an LLM agent reads -bytes, not pixels — or (b) narrows to the whitespace/look-alike -branches and recommends the host-side byte-compare (the PowerShell -snippet from the playbook) to identify the code point. The agent -must NOT assert a specific code point without byte-level evidence, -and must NOT pick a branch (typo, case, rename) the CLI evidence -rules out. - -**The branches that COULD apply** without further evidence: - -- Leading/trailing whitespace (branch 5) — visible only when - inspecting byte-by-byte. -- Look-alike Unicode character (branch 6) — NBSP, Cyrillic vs - Latin letters, curly vs straight quotes; visually identical in - every UI surface. - -**The branches that are ruled OUT by the CLI evidence:** - -- Typo (branch 1) — configured and logged names are visually - identical. -- Case mismatch (branch 2) — visually identical, same case. -- Sheet renamed (branch 3) — the configured name appears in the - Get Workbook Sheets log output. -- Sheet deleted (branch 4) — the configured name appears in the - Get Workbook Sheets log output. -- Dynamic expression resolved wrong (branch 7) — workflow source - shows a literal `SheetName: "Quarterly Data"`. - -The agent may resolve between the remaining branches by inspecting -the bytes the job-log evidence already carries; what it must NOT do -is assert a specific code point it never inspected. Unsupported -confident guessing is the failure mode this test catches. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` → `Get Workbook Sheets` (logs result) → `Read Range "Quarterly Data"` (fails). Configured name uses regular space `U+0020` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses. The job logs surface the actual sheet name with `U+00A0` between the words; visible only via byte-inspection | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` → `jobs logs` → workflow -source review → recognize that all visible names match → identify -the NBSP from the log bytes, or recommend the host-side -byte-compare command from the playbook. - -> **Note on fixtures.** Synthetic. The actual sheet name in the -> fixture log message is stored as a real non-breaking space byte -> in the mock response. It renders identically to a regular space -> for a human reader, but the bytes reach the agent's tool output -> intact — so byte-level identification is a legitimate diagnosis -> path. Keep the raw NBSP byte in the mock (a six-character -> backslash-u escape would disclose the answer as plain ASCII). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/d35bb9b790.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/d35bb9b790.json deleted file mode 100644 index f2fe556ff9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/d35bb9b790.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-7777-8888-9999-000011112222", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9cc33333-7777-8888-9999-000011112222", - "Info": "UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist.\r\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\r\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-19T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelReadRangeX_1", - "ActivityDisplayName": "Read Range: Quarterly Data!A1", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-7777-8888-9999-000011112222", - "Id": 680066001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/manifest.json deleted file mode 100644 index 28a8a2e08e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/m/r/manifest.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "version": 2, - "_doc": "Canned uip responses for investigating a faulted Excel Read Range job (folders, jobs, job logs).", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "6b808e9e6a.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "6b808e9e6a.json" - }, - { - "match": "or jobs get \"cc333333-7777-8888-9999-000011112222\" --output json", - "file": "d35bb9b790.json" - }, - { - "match": "or jobs get cc333333-7777-8888-9999-000011112222 --output json", - "file": "d35bb9b790.json" - }, - { - "match": "or jobs history \"cc333333-7777-8888-9999-000011112222\" --output json", - "file": "61d7a8b016.json" - }, - { - "match": "or jobs history cc333333-7777-8888-9999-000011112222 --output json", - "file": "61d7a8b016.json" - }, - { - "match": "or jobs logs \"cc333333-7777-8888-9999-000011112222\" --output json", - "file": "22546d1c6d.json" - }, - { - "match": "or jobs logs cc333333-7777-8888-9999-000011112222 --output json", - "file": "22546d1c6d.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/uip-fixture.json new file mode 100644 index 0000000000..f9634bc7ea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cc333333-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9cc33333-7777-8888-9999-000011112222\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist.\\r\\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\\r\\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelReadRangeX_1\",\r\n \"ActivityDisplayName\": \"Read Range: Quarterly Data!A1\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Id\": 680066001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cc333333-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cc333333-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc333377-001\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333377-002\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333377-003\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333377-004\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.612Z\",\r\n \"Message\": \"Get Workbook Sheets — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333377-005\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.812Z\",\r\n \"Message\": \"[ExcelDailyImport] Available sheets in workbook: [\\\"Sheet1\\\", \\\"Quarterly Data\\\", \\\"Summary\\\"]\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333377-006\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.512Z\",\r\n \"Message\": \"Read Range: Quarterly Data!A1 — executing (SheetName='Quarterly Data')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333377-007\",\r\n \"JobKey\": \"cc333333-7777-8888-9999-000011112222\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.802Z\",\r\n \"Message\": \"Read Range: Quarterly Data!A1 — UiPath.Excel.BusinessException: The sheet with the name 'Quarterly Data' does not exist.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/task.yaml index 5f94be1cbf..c98fe14bf4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-bytes/task.yaml @@ -26,21 +26,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/README.md new file mode 100644 index 0000000000..ff64d85e5f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/README.md @@ -0,0 +1,57 @@ +# Excel Read Range — Sheet Name Case Mismatch on OpenXML + +This scenario reproduces a Read Range failure where the configured +`SheetName` differs from the actual sheet only in letter casing, +and the activity runs on the OpenXML provider (case-sensitive). The +job ends with: + +``` +UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist. +``` + +## What this scenario uncovers + +**Root Cause:** Workflow configured `SheetName: "data"` (lowercase) +on the Read Range activity. The workbook's actual sheet is `Data` +(capital D). The workflow uses `Use Excel File` (modern scope) with +no `ReadFormatting`, `EditPassword`, or macro-related properties — +the runtime uses the OpenXML provider, which is **case-sensitive** +in sheet name lookups. The agent must: + +1. Match the playbook (BusinessException fingerprint). +2. Read the job logs and find the `Get Workbook Sheets` output + showing `["Sheet1", "Data", "Summary"]`. +3. Read workflow source and see that the scope is `Use Excel File` + with no COM-forcing properties. +4. Conclude: case-mismatch branch — `"data"` vs `"Data"` only + differs in case, and the active provider (OpenXML) is + case-sensitive. +5. Explain the user's "worked yesterday on my dev machine" clue: + the dev machine has Excel installed → COM provider → + case-insensitive; the Robot host has no Excel → OpenXML → + case-sensitive. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-sheet-not-found.md` +(the "Case mismatch under the OpenXML provider" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` (no COM-forcing properties) → `Get Workbook Sheets` (logs result) → `Read Range "data"` (fails) | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` → `jobs logs` (reveals +actual sheets list) → workflow source inspection (identifies +`Use Excel File` scope + no COM-forcing properties → OpenXML +provider). + +> **Note on fixtures.** Synthetic. The job key, folder key, +> workbook path are placeholders — the test grades whether the +> agent surfaces the case-mismatch branch with both the casing +> evidence AND the provider-behavior reasoning. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/0ac7b4bac4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/0ac7b4bac4.json deleted file mode 100644 index e5008d40a4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/0ac7b4bac4.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-bb222277-001", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222277-002", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222277-003", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — executing (provider: OpenXML, Excel COM not available on host)", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222277-004", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.612Z", - "Message": "Get Workbook Sheets — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222277-005", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.812Z", - "Message": "[ExcelDailyImport] Available sheets in workbook: [\"Sheet1\", \"Data\", \"Summary\"]", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222277-006", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:02.512Z", - "Message": "Read Range: data!A1 — executing (SheetName='data')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222277-007", - "JobKey": "bb222222-7777-8888-9999-000011112222", - "Level": "Error", - "Timestamp": "2026-05-19T08:00:02.802Z", - "Message": "Read Range: data!A1 — UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/5b1ed2d2b7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/5b1ed2d2b7.json deleted file mode 100644 index 11e8b04907..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/5b1ed2d2b7.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "bb222222-7777-8888-9999-000011112222", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/6ab339f308.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/6ab339f308.json deleted file mode 100644 index 7fc8cce4e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/6ab339f308.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "bb222222-7777-8888-9999-000011112222", - "State": "Pending", - "Timestamp": "2026-05-19T08:00:01.005Z" - }, - { - "Key": "bb222222-7777-8888-9999-000011112222", - "State": "Running", - "Timestamp": "2026-05-19T08:00:01.300Z" - }, - { - "Key": "bb222222-7777-8888-9999-000011112222", - "State": "Faulted", - "Timestamp": "2026-05-19T08:00:02.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/8216031ee4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/8216031ee4.json deleted file mode 100644 index 123c2f546f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/8216031ee4.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-7777-8888-9999-000011112222", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-7777-8888-9999-000011112222", - "Info": "UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist.\r\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\r\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-19T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelReadRangeX_1", - "ActivityDisplayName": "Read Range: data!A1", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "bb222222-7777-8888-9999-000011112222", - "Id": 680055001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/README.md deleted file mode 100644 index fc60b27386..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Excel Read Range — Sheet Name Case Mismatch on OpenXML - -This scenario reproduces a Read Range failure where the configured -`SheetName` differs from the actual sheet only in letter casing, -and the activity runs on the OpenXML provider (case-sensitive). The -job ends with: - -``` -UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist. -``` - -## What this scenario uncovers - -**Root Cause:** Workflow configured `SheetName: "data"` (lowercase) -on the Read Range activity. The workbook's actual sheet is `Data` -(capital D). The workflow uses `Use Excel File` (modern scope) with -no `ReadFormatting`, `EditPassword`, or macro-related properties — -the runtime uses the OpenXML provider, which is **case-sensitive** -in sheet name lookups. The agent must: - -1. Match the playbook (BusinessException fingerprint). -2. Read the job logs and find the `Get Workbook Sheets` output - showing `["Sheet1", "Data", "Summary"]`. -3. Read workflow source and see that the scope is `Use Excel File` - with no COM-forcing properties. -4. Conclude: case-mismatch branch — `"data"` vs `"Data"` only - differs in case, and the active provider (OpenXML) is - case-sensitive. -5. Explain the user's "worked yesterday on my dev machine" clue: - the dev machine has Excel installed → COM provider → - case-insensitive; the Robot host has no Excel → OpenXML → - case-sensitive. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-sheet-not-found.md` -(the "Case mismatch under the OpenXML provider" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` (no COM-forcing properties) → `Get Workbook Sheets` (logs result) → `Read Range "data"` (fails) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` → `jobs logs` (reveals -actual sheets list) → workflow source inspection (identifies -`Use Excel File` scope + no COM-forcing properties → OpenXML -provider). - -> **Note on fixtures.** Synthetic. The job key, folder key, -> workbook path are placeholders — the test grades whether the -> agent surfaces the case-mismatch branch with both the casing -> evidence AND the provider-behavior reasoning. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/manifest.json deleted file mode 100644 index 9cbf447535..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/m/r/manifest.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range sheet-not-found scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "5b1ed2d2b7.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "5b1ed2d2b7.json" - }, - { - "match": "or jobs get \"bb222222-7777-8888-9999-000011112222\" --output json", - "file": "8216031ee4.json" - }, - { - "match": "or jobs get bb222222-7777-8888-9999-000011112222 --output json", - "file": "8216031ee4.json" - }, - { - "match": "or jobs history \"bb222222-7777-8888-9999-000011112222\" --output json", - "file": "6ab339f308.json" - }, - { - "match": "or jobs history bb222222-7777-8888-9999-000011112222 --output json", - "file": "6ab339f308.json" - }, - { - "match": "or jobs logs \"bb222222-7777-8888-9999-000011112222\" --output json", - "file": "0ac7b4bac4.json" - }, - { - "match": "or jobs logs bb222222-7777-8888-9999-000011112222 --output json", - "file": "0ac7b4bac4.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/uip-fixture.json new file mode 100644 index 0000000000..35b684e128 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb222222-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-7777-8888-9999-000011112222\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist.\\r\\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\\r\\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelReadRangeX_1\",\r\n \"ActivityDisplayName\": \"Read Range: data!A1\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Id\": 680055001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "bb222222-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb222222-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-bb222277-001\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222277-002\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222277-003\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — executing (provider: OpenXML, Excel COM not available on host)\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222277-004\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.612Z\",\r\n \"Message\": \"Get Workbook Sheets — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222277-005\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.812Z\",\r\n \"Message\": \"[ExcelDailyImport] Available sheets in workbook: [\\\"Sheet1\\\", \\\"Data\\\", \\\"Summary\\\"]\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222277-006\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.512Z\",\r\n \"Message\": \"Read Range: data!A1 — executing (SheetName='data')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222277-007\",\r\n \"JobKey\": \"bb222222-7777-8888-9999-000011112222\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.802Z\",\r\n \"Message\": \"Read Range: data!A1 — UiPath.Excel.BusinessException: The sheet with the name 'data' does not exist.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/task.yaml index 58efdabd4c..c1a663d8b2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-case/task.yaml @@ -25,21 +25,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/README.md new file mode 100644 index 0000000000..dc8ed08a16 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/README.md @@ -0,0 +1,44 @@ +# Excel Read Range — Dynamic SheetName Resolved To Empty At Runtime + +Runtime troubleshooting scenario for `UiPath.Excel.Activities` Read Range (`ExcelReadRangeX`), covering +the **`read-range-sheet-not-found.md` branch 7** — a dynamic `SheetName` expression that resolved to the +wrong value (here, an empty string) at runtime. + +## What this scenario exercises + +An unattended job faults with `UiPath.Excel.BusinessException: The sheet with the name '' does not +exist.` The workbook opened fine and `Get Workbook Sheets` returned the real sheet list. The agent must +recognize that the `SheetName` is **not a literal** but the dynamic expression `[in_TargetSheet]`, that +the input argument arrived **empty** (`InputArguments: {"in_TargetSheet":""}`, log `Target sheet +resolved to: ''`), and prescribe fixing the upstream source that feeds the argument plus adding a +validate/guard before Read Range. It must NOT diagnose a typo, rename, whitespace, look-alike-character, +or hidden-sheet cause — there is no configured literal to correct. + +This is the distinguishing skill for branch 7: the error names a value nobody configured (`''`), so the +fault is upstream data, not a mistyped sheet name. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted modern VB project: `Main.xaml` with `Use Excel File` → `Get Workbook Sheets` → Read Range whose `SheetName` is `[in_TargetSheet]`; the project declares the `in_TargetSheet` input argument | + +The diagnosis is not disclosed in any agent-visible name: the project is `MonthlySalesImport`, the activity +is `Read monthly sales sheet`. The prompt states the observed symptom (the pasted error with the empty +`''` name and the job key); the cause (empty input argument) is derived from the source + job evidence. + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the dynamic `SheetName` bound to the empty `in_TargetSheet` argument as the cause, + and the fix (correct the upstream source that supplies the argument + validate the resolved sheet name + against Get Workbook Sheets before reading) — not a typo/rename/whitespace/hidden-sheet misdiagnosis. + +Playbook: `references/activity-packages/excel-activities/playbooks/read-range-sheet-not-found.md` +(branch 7 — variable resolved to wrong value). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/244ab94cfe.json deleted file mode 100644 index 481b5c95ba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "bb222222-8888-9999-0000-111122223333", - "State": "Pending", - "Timestamp": "2026-05-20T06:00:00.900Z" - }, - { - "Key": "bb222222-8888-9999-0000-111122223333", - "State": "Running", - "Timestamp": "2026-05-20T06:00:01.240Z" - }, - { - "Key": "bb222222-8888-9999-0000-111122223333", - "State": "Faulted", - "Timestamp": "2026-05-20T06:00:02.690Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/464db476bb.json deleted file mode 100644 index 9eca9529cc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/464db476bb.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-8888-9999-0000-111122223333", - "StartTime": "2026-05-20T06:00:01.240Z", - "EndTime": "2026-05-20T06:00:02.690Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-8888-9999-0000-111122223333", - "Info": "UiPath.Excel.BusinessException: The sheet with the name '' does not exist.\r\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\r\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T06:00:00.900Z", - "ReleaseName": "MonthlySalesImport", - "Type": "Unattended", - "InputArguments": "{\"in_TargetSheet\":\"\"}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5690101, - "OrganizationUnitId": 6020001, - "FolderKey": "f0022222-3333-4444-5555-666677778888", - "Reference": "MonthlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T06:00:02.690Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelReadRangeX_1", - "ActivityDisplayName": "Read monthly sales sheet", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "bb222222-8888-9999-0000-111122223333", - "Id": 690055001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/6059da6ae1.json deleted file mode 100644 index ead3da5a7a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "bb222222-8888-9999-0000-111122223333", - "StartTime": "2026-05-20T06:00:01.240Z", - "EndTime": "2026-05-20T06:00:02.690Z", - "State": "Faulted", - "ReleaseName": "MonthlySalesImport", - "Source": "Schedule", - "Reference": "MonthlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022222-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: The sheet with the name '' does not exist.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/78f94b6048.json deleted file mode 100644 index d0dccff213..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/78f94b6048.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022222-3333-4444-5555-666677778888", - "DisplayName": "SalesReporting", - "FullyQualifiedName": "SalesReporting", - "Description": "Folder for scheduled sales reporting processes.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/README.md deleted file mode 100644 index 5451bb3183..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Excel Read Range — Dynamic SheetName Resolved To Empty At Runtime - -Runtime troubleshooting scenario for `UiPath.Excel.Activities` Read Range (`ExcelReadRangeX`), covering -the **`read-range-sheet-not-found.md` branch 7** — a dynamic `SheetName` expression that resolved to the -wrong value (here, an empty string) at runtime. - -## What this scenario exercises - -An unattended job faults with `UiPath.Excel.BusinessException: The sheet with the name '' does not -exist.` The workbook opened fine and `Get Workbook Sheets` returned the real sheet list. The agent must -recognize that the `SheetName` is **not a literal** but the dynamic expression `[in_TargetSheet]`, that -the input argument arrived **empty** (`InputArguments: {"in_TargetSheet":""}`, log `Target sheet -resolved to: ''`), and prescribe fixing the upstream source that feeds the argument plus adding a -validate/guard before Read Range. It must NOT diagnose a typo, rename, whitespace, look-alike-character, -or hidden-sheet cause — there is no configured literal to correct. - -This is the distinguishing skill for branch 7: the error names a value nobody configured (`''`), so the -fault is upstream data, not a mistyped sheet name. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted modern VB project: `Main.xaml` with `Use Excel File` → `Get Workbook Sheets` → Read Range whose `SheetName` is `[in_TargetSheet]`; the project declares the `in_TargetSheet` input argument | -| `data/m/r/` | canned `Faulted` job; `job-get.json` has `InputArguments {"in_TargetSheet":""}` and a `BusinessException` for sheet `''`; `job-logs.json` shows actual sheets `["Sheet1","Data","Summary"]` and `Target sheet resolved to: ''`; `docsai ask` passthrough | - -The diagnosis is not leaked in any agent-visible name: the project is `MonthlySalesImport`, the activity -is `Read monthly sales sheet`. The prompt states the observed symptom (the pasted error with the empty -`''` name and the job key); the cause (empty input argument) is derived from the source + job evidence. - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the dynamic `SheetName` bound to the empty `in_TargetSheet` argument as the cause, - and the fix (correct the upstream source that supplies the argument + validate the resolved sheet name - against Get Workbook Sheets before reading) — not a typo/rename/whitespace/hidden-sheet misdiagnosis. - -Playbook: `references/activity-packages/excel-activities/playbooks/read-range-sheet-not-found.md` -(branch 7 — variable resolved to wrong value). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/e2a5346a88.json deleted file mode 100644 index 93d2d7e7ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-bb222288-001", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Info", - "Timestamp": "2026-05-20T06:00:01.250Z", - "Message": "Execution started.", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-002", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Info", - "Timestamp": "2026-05-20T06:00:01.360Z", - "Message": "[MonthlySalesImport] Starting monthly sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-003", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Trace", - "Timestamp": "2026-05-20T06:00:01.470Z", - "Message": "Use Excel File: sales-2026-05.xlsx — executing", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-004", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Trace", - "Timestamp": "2026-05-20T06:00:01.560Z", - "Message": "Get Workbook Sheets — executing", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-005", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Info", - "Timestamp": "2026-05-20T06:00:01.760Z", - "Message": "[MonthlySalesImport] Available sheets in workbook: [\"Sheet1\", \"Data\", \"Summary\"]", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-006", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Info", - "Timestamp": "2026-05-20T06:00:02.010Z", - "Message": "[MonthlySalesImport] Target sheet resolved to: ''", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-007", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Trace", - "Timestamp": "2026-05-20T06:00:02.410Z", - "Message": "Read monthly sales sheet — executing (SheetName='')", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222288-008", - "JobKey": "bb222222-8888-9999-0000-111122223333", - "Level": "Error", - "Timestamp": "2026-05-20T06:00:02.680Z", - "Message": "Read monthly sales sheet — UiPath.Excel.BusinessException: The sheet with the name '' does not exist.", - "ProcessName": "MonthlySalesImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/manifest.json deleted file mode 100644 index 87345d5f06..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/m/r/manifest.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range sheet-not-found (dynamic SheetName resolved empty) scenario; mock dispatcher only - do not treat as investigation evidence. docsai proxied to the real CLI; job/folder lookups return a canned Faulted-job fixture. Matching is token-aware (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/uip-fixture.json new file mode 100644 index 0000000000..351acfa4d4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022222-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"SalesReporting\",\r\n \"FullyQualifiedName\": \"SalesReporting\",\r\n \"Description\": \"Folder for scheduled sales reporting processes.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"StartTime\": \"2026-05-20T06:00:01.240Z\",\r\n \"EndTime\": \"2026-05-20T06:00:02.690Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"MonthlySalesImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"MonthlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022222-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name '' does not exist.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-bb222288-001\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T06:00:01.250Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-002\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T06:00:01.360Z\",\r\n \"Message\": \"[MonthlySalesImport] Starting monthly sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-003\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T06:00:01.470Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — executing\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-004\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T06:00:01.560Z\",\r\n \"Message\": \"Get Workbook Sheets — executing\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-005\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T06:00:01.760Z\",\r\n \"Message\": \"[MonthlySalesImport] Available sheets in workbook: [\\\"Sheet1\\\", \\\"Data\\\", \\\"Summary\\\"]\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-006\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T06:00:02.010Z\",\r\n \"Message\": \"[MonthlySalesImport] Target sheet resolved to: ''\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-007\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T06:00:02.410Z\",\r\n \"Message\": \"Read monthly sales sheet — executing (SheetName='')\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222288-008\",\r\n \"JobKey\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T06:00:02.680Z\",\r\n \"Message\": \"Read monthly sales sheet — UiPath.Excel.BusinessException: The sheet with the name '' does not exist.\",\r\n \"ProcessName\": \"MonthlySalesImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-20T06:00:00.900Z\"\r\n },\r\n {\r\n \"Key\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-20T06:00:01.240Z\"\r\n },\r\n {\r\n \"Key\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-20T06:00:02.690Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"StartTime\": \"2026-05-20T06:00:01.240Z\",\r\n \"EndTime\": \"2026-05-20T06:00:02.690Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-8888-9999-0000-111122223333\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name '' does not exist.\\r\\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\\r\\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T06:00:00.900Z\",\r\n \"ReleaseName\": \"MonthlySalesImport\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{\\\"in_TargetSheet\\\":\\\"\\\"}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5690101,\r\n \"OrganizationUnitId\": 6020001,\r\n \"FolderKey\": \"f0022222-3333-4444-5555-666677778888\",\r\n \"Reference\": \"MonthlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T06:00:02.690Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelReadRangeX_1\",\r\n \"ActivityDisplayName\": \"Read monthly sales sheet\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"bb222222-8888-9999-0000-111122223333\",\r\n \"Id\": 690055001\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/task.yaml index 6b94f102b8..0c63527e7b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-dynamic-empty/task.yaml @@ -21,21 +21,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/README.md new file mode 100644 index 0000000000..5fecdb6c5e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/README.md @@ -0,0 +1,45 @@ +# Excel Read Range — Sheet Name Typo + +This scenario reproduces a Read Range failure where the configured +`SheetName` is a one-character typo of an actual sheet in the +workbook. The job ends with: + +``` +UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist. +``` + +## What this scenario uncovers + +**Root Cause:** Workflow configured `SheetName: "Datab"` on the +`ExcelReadRange` activity. The workbook's actual sheets are +`["Sheet1", "Data", "Summary"]`. The configured name is a typo of +`Data` (extra trailing `b`). + +The workflow's `Get Workbook Sheets` activity runs first, succeeds, +and logs the actual sheet titles via `LogMessage`. The agent reads +the job logs, finds the enumerated list, compares against the +configured name, and identifies the typo. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-sheet-not-found.md` +(the "Typo in the configured sheet name" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal `ExcelDailyImport` project — `Use Excel File` → `Get Workbook Sheets` (logs result) → `Read Range "Datab"` (fails) | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (current) → `jobs logs` +(reveals the enumerated sheet list from the prior `Get Workbook +Sheets` activity) → workflow source inspection → conclude typo. + +> **Note on fixtures.** Synthetic. Job keys, folder key, workbook +> path are placeholders — the test grades whether the agent surfaces +> the typo branch with the actual-sheets evidence, not the specific +> names. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/1d986c3977.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/1d986c3977.json deleted file mode 100644 index cbf4a35879..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/1d986c3977.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "aa111111-7777-8888-9999-000011112222", - "State": "Pending", - "Timestamp": "2026-05-19T08:00:01.005Z" - }, - { - "Key": "aa111111-7777-8888-9999-000011112222", - "State": "Running", - "Timestamp": "2026-05-19T08:00:01.300Z" - }, - { - "Key": "aa111111-7777-8888-9999-000011112222", - "State": "Faulted", - "Timestamp": "2026-05-19T08:00:02.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/37b3818139.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/37b3818139.json deleted file mode 100644 index b81a470262..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/37b3818139.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-7777-8888-9999-000011112222", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/758a37b323.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/758a37b323.json deleted file mode 100644 index 313a441320..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/758a37b323.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-7777-8888-9999-000011112222", - "StartTime": "2026-05-19T08:00:01.300Z", - "EndTime": "2026-05-19T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-7777-8888-9999-000011112222", - "Info": "UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist.\r\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\r\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-19T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelReadRangeX_1", - "ActivityDisplayName": "Read Range: Datab!A1", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-7777-8888-9999-000011112222", - "Id": 680044001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/README.md deleted file mode 100644 index 540a703092..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Excel Read Range — Sheet Name Typo - -This scenario reproduces a Read Range failure where the configured -`SheetName` is a one-character typo of an actual sheet in the -workbook. The job ends with: - -``` -UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist. -``` - -## What this scenario uncovers - -**Root Cause:** Workflow configured `SheetName: "Datab"` on the -`ExcelReadRange` activity. The workbook's actual sheets are -`["Sheet1", "Data", "Summary"]`. The configured name is a typo of -`Data` (extra trailing `b`). - -The workflow's `Get Workbook Sheets` activity runs first, succeeds, -and logs the actual sheet titles via `LogMessage`. The agent reads -the job logs, finds the enumerated list, compares against the -configured name, and identifies the typo. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-sheet-not-found.md` -(the "Typo in the configured sheet name" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | minimal `ExcelDailyImport` project — `Use Excel File` → `Get Workbook Sheets` (logs result) → `Read Range "Datab"` (fails) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (current) → `jobs logs` -(reveals the enumerated sheet list from the prior `Get Workbook -Sheets` activity) → workflow source inspection → conclude typo. - -> **Note on fixtures.** Synthetic. Job keys, folder key, workbook -> path are placeholders — the test grades whether the agent surfaces -> the typo branch with the actual-sheets evidence, not the specific -> names. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/d8080423b7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/d8080423b7.json deleted file mode 100644 index 21ecad9ca5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/d8080423b7.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa111177-001", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111177-002", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111177-003", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111177-004", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:01.612Z", - "Message": "Get Workbook Sheets — executing", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111177-005", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Info", - "Timestamp": "2026-05-19T08:00:01.812Z", - "Message": "[ExcelDailyImport] Available sheets in workbook: [\"Sheet1\", \"Data\", \"Summary\"]", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111177-006", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Trace", - "Timestamp": "2026-05-19T08:00:02.512Z", - "Message": "Read Range: Datab!A1 — executing (SheetName='Datab')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111177-007", - "JobKey": "aa111111-7777-8888-9999-000011112222", - "Level": "Error", - "Timestamp": "2026-05-19T08:00:02.802Z", - "Message": "Read Range: Datab!A1 — UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/manifest.json deleted file mode 100644 index ae1f7ac241..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/m/r/manifest.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range sheet-not-found scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "37b3818139.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "37b3818139.json" - }, - { - "match": "or jobs get \"aa111111-7777-8888-9999-000011112222\" --output json", - "file": "758a37b323.json" - }, - { - "match": "or jobs get aa111111-7777-8888-9999-000011112222 --output json", - "file": "758a37b323.json" - }, - { - "match": "or jobs history \"aa111111-7777-8888-9999-000011112222\" --output json", - "file": "1d986c3977.json" - }, - { - "match": "or jobs history aa111111-7777-8888-9999-000011112222 --output json", - "file": "1d986c3977.json" - }, - { - "match": "or jobs logs \"aa111111-7777-8888-9999-000011112222\" --output json", - "file": "d8080423b7.json" - }, - { - "match": "or jobs logs aa111111-7777-8888-9999-000011112222 --output json", - "file": "d8080423b7.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/uip-fixture.json new file mode 100644 index 0000000000..0ea56ccdc1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa111111-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"StartTime\": \"2026-05-19T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-7777-8888-9999-000011112222\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist.\\r\\n at UiPath.Excel.WorkbookActivities.ReadRangeImpl.ResolveSheet(IExcelWorkbook workbook, String sheetName)\\r\\n at UiPath.Excel.Activities.ExcelReadRangeX.ExecuteAsync(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelReadRangeX_1\",\r\n \"ActivityDisplayName\": \"Read Range: Datab!A1\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Id\": 680044001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa111111-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa111111-7777-8888-9999-000011112222", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa111177-001\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111177-002\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111177-003\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111177-004\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.612Z\",\r\n \"Message\": \"Get Workbook Sheets — executing\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111177-005\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T08:00:01.812Z\",\r\n \"Message\": \"[ExcelDailyImport] Available sheets in workbook: [\\\"Sheet1\\\", \\\"Data\\\", \\\"Summary\\\"]\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111177-006\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.512Z\",\r\n \"Message\": \"Read Range: Datab!A1 — executing (SheetName='Datab')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111177-007\",\r\n \"JobKey\": \"aa111111-7777-8888-9999-000011112222\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T08:00:02.802Z\",\r\n \"Message\": \"Read Range: Datab!A1 — UiPath.Excel.BusinessException: The sheet with the name 'Datab' does not exist.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/task.yaml index 7b627e1d0f..2eb25a614e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-sheet-typo/task.yaml @@ -24,21 +24,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/README.md new file mode 100644 index 0000000000..10da44631f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/README.md @@ -0,0 +1,53 @@ +# Excel Read Range — Mapped Drive Not Mapped Under Robot Session + +This scenario reproduces a Read Range failure where the workflow uses +a mapped drive path (`Z:\Data\sales-2026-05.xlsx`) that exists in the +developer's interactive RDP session but is **not** mapped in the +Robot service's unattended session on the same host. The job ends +with: + +``` +System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\Data\sales-2026-05.xlsx'. +``` + +## What this scenario uncovers + +**Root Cause:** Windows mapped drives are **per-session**. A drive +letter mapped via `net use` (or via Explorer's "Map Network Drive") +in an interactive user session is visible to that session only. The +Robot service runs in its own session and does not inherit the +interactive user's drive mappings, even on the same host. The user's +"works on my machine" experience (when they RDP in and see +`Z:\Data\...`) is misleading. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-file-not-found.md` +(the "Drive letter not mapped under Robot session" branch). + +The CLI evidence chain (workflow source `Z:\` prefix + +`DirectoryNotFoundException` on the drive root) gives the agent a +strong indicator, but verifying the drive mappings on the host +requires either confident reasoning about per-session drive +mappings OR a host-side `net use` check under the Robot's session. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDailyImport` project — `Use Excel File` with **mapped-drive** `WorkbookPath: "Z:\Data\sales-2026-05.xlsx"`, then `Read Range` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (sees `Z:\` prefix + +DirectoryNotFoundException on drive root) → `jobs logs` → workflow +source review (literal `Z:\` mapped-drive path) → either (A) +confident branch 4 diagnosis or (B) recommend `net use` host-side +check. + +> **Note on fixtures.** Synthetic. The job key, folder key, and +> mapped-drive letter are placeholders. The test grades whether +> the agent surfaces the per-session-drive-mapping pattern using +> the workflow-source evidence + reasoning. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/372fef8cc1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/372fef8cc1.json deleted file mode 100644 index 03435de09d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/372fef8cc1.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ee55555-aaaa-bbbb-cccc-dddddddddddd", - "Info": "System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\\Data\\sales-2026-05.xlsx'.\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, ...)\r\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\r\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelDailyImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680101, - "OrganizationUnitId": 6010001, - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Reference": "HourlySalesA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:01.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "DirectoryNotFoundException", - "ActivityName": "UseExcelFile_1", - "ActivityDisplayName": "Use Excel File: Z:\\Data\\sales-2026-05.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "Id": 680077005 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/4789cfae97.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/4789cfae97.json deleted file mode 100644 index a3f51e8a48..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/4789cfae97.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ee5555aa-001", - "JobKey": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee5555aa-002", - "JobKey": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelDailyImport] Starting daily sales import — reading Z:\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee5555aa-003", - "JobKey": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: Z:\\Data\\sales-2026-05.xlsx — opening workbook (WorkbookPath='Z:\\Data\\sales-2026-05.xlsx')", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee5555aa-004", - "JobKey": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:01.802Z", - "Message": "Use Excel File: Z:\\Data\\sales-2026-05.xlsx — System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\\Data\\sales-2026-05.xlsx'.", - "ProcessName": "ExcelDailyImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/6774b2d6cd.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/6774b2d6cd.json deleted file mode 100644 index a5cda880a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/6774b2d6cd.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:01.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDailyImport", - "Source": "Schedule", - "Reference": "HourlySalesA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011111-2222-3333-4444-555566667777", - "Info": "System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\\Data\\sales-2026-05.xlsx'.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/858738168e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/858738168e.json deleted file mode 100644 index 802096e9ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/858738168e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011111-2222-3333-4444-555566667777", - "DisplayName": "ExcelImports", - "FullyQualifiedName": "ExcelImports", - "Description": "Folder for daily Excel imports.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6010001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/README.md deleted file mode 100644 index 20a3199715..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# Excel Read Range — Mapped Drive Not Mapped Under Robot Session - -This scenario reproduces a Read Range failure where the workflow uses -a mapped drive path (`Z:\Data\sales-2026-05.xlsx`) that exists in the -developer's interactive RDP session but is **not** mapped in the -Robot service's unattended session on the same host. The job ends -with: - -``` -System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\Data\sales-2026-05.xlsx'. -``` - -## What this scenario uncovers - -**Root Cause:** Windows mapped drives are **per-session**. A drive -letter mapped via `net use` (or via Explorer's "Map Network Drive") -in an interactive user session is visible to that session only. The -Robot service runs in its own session and does not inherit the -interactive user's drive mappings, even on the same host. The user's -"works on my machine" experience (when they RDP in and see -`Z:\Data\...`) is misleading. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/read-range-file-not-found.md` -(the "Drive letter not mapped under Robot session" branch). - -The CLI evidence chain (workflow source `Z:\` prefix + -`DirectoryNotFoundException` on the drive root) gives the agent a -strong indicator, but verifying the drive mappings on the host -requires either confident reasoning about per-session drive -mappings OR a host-side `net use` check under the Robot's session. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDailyImport` project — `Use Excel File` with **mapped-drive** `WorkbookPath: "Z:\Data\sales-2026-05.xlsx"`, then `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field shows the drive-root `DirectoryNotFoundException` | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (sees `Z:\` prefix + -DirectoryNotFoundException on drive root) → `jobs logs` → workflow -source review (literal `Z:\` mapped-drive path) → either (A) -confident branch 4 diagnosis or (B) recommend `net use` host-side -check. - -> **Note on fixtures.** Synthetic. The job key, folder key, and -> mapped-drive letter are placeholders. The test grades whether -> the agent surfaces the per-session-drive-mapping pattern using -> the workflow-source evidence + reasoning. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/a77f0a54d6.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/a77f0a54d6.json deleted file mode 100644 index 5fab05d7a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/a77f0a54d6.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "State": "Pending", - "Timestamp": "2026-05-20T08:00:01.005Z" - }, - { - "Key": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "State": "Running", - "Timestamp": "2026-05-20T08:00:01.300Z" - }, - { - "Key": "ee555555-aaaa-bbbb-cccc-dddddddddddd", - "State": "Faulted", - "Timestamp": "2026-05-20T08:00:01.812Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/manifest.json deleted file mode 100644 index 551334703a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/m/r/manifest.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Read Range file-not-found scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or folders list --output json", - "file": "858738168e.json" - }, - { - "match": "or jobs list --folder-key \"f0011111-2222-3333-4444-555566667777\" --state Faulted --output json", - "file": "6774b2d6cd.json" - }, - { - "match": "or jobs list --folder-key f0011111-2222-3333-4444-555566667777 --state Faulted --output json", - "file": "6774b2d6cd.json" - }, - { - "match": "or jobs get \"ee555555-aaaa-bbbb-cccc-dddddddddddd\" --output json", - "file": "372fef8cc1.json" - }, - { - "match": "or jobs get ee555555-aaaa-bbbb-cccc-dddddddddddd --output json", - "file": "372fef8cc1.json" - }, - { - "match": "or jobs history \"ee555555-aaaa-bbbb-cccc-dddddddddddd\" --output json", - "file": "a77f0a54d6.json" - }, - { - "match": "or jobs history ee555555-aaaa-bbbb-cccc-dddddddddddd --output json", - "file": "a77f0a54d6.json" - }, - { - "match": "or jobs logs \"ee555555-aaaa-bbbb-cccc-dddddddddddd\" --output json", - "file": "4789cfae97.json" - }, - { - "match": "or jobs logs ee555555-aaaa-bbbb-cccc-dddddddddddd --output json", - "file": "4789cfae97.json" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/uip-fixture.json new file mode 100644 index 0000000000..6529d8de79 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0011111-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ee555555-aaaa-bbbb-cccc-dddddddddddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ee55555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"Info\": \"System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\\\\Data\\\\sales-2026-05.xlsx'.\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, ...)\\r\\n at UiPath.Excel.WorkbookOpener.OpenAsync(String path, OpenMode mode, CancellationToken cancellationToken)\\r\\n at UiPath.Excel.Activities.UseExcelFile.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDailyImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680101,\r\n \"OrganizationUnitId\": 6010001,\r\n \"FolderKey\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HourlySalesA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:01.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"DirectoryNotFoundException\",\r\n \"ActivityName\": \"UseExcelFile_1\",\r\n \"ActivityDisplayName\": \"Use Excel File: Z:\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"Id\": 680077005\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ee555555-aaaa-bbbb-cccc-dddddddddddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.005Z\"\r\n },\r\n {\r\n \"Key\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.300Z\"\r\n },\r\n {\r\n \"Key\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.812Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ee555555-aaaa-bbbb-cccc-dddddddddddd", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ee5555aa-001\",\r\n \"JobKey\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee5555aa-002\",\r\n \"JobKey\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDailyImport] Starting daily sales import — reading Z:\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee5555aa-003\",\r\n \"JobKey\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: Z:\\\\Data\\\\sales-2026-05.xlsx — opening workbook (WorkbookPath='Z:\\\\Data\\\\sales-2026-05.xlsx')\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee5555aa-004\",\r\n \"JobKey\": \"ee555555-aaaa-bbbb-cccc-dddddddddddd\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.802Z\",\r\n \"Message\": \"Use Excel File: Z:\\\\Data\\\\sales-2026-05.xlsx — System.IO.DirectoryNotFoundException: Could not find a part of the path 'Z:\\\\Data\\\\sales-2026-05.xlsx'.\",\r\n \"ProcessName\": \"ExcelDailyImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelImports\",\r\n \"FullyQualifiedName\": \"ExcelImports\",\r\n \"Description\": \"Folder for daily Excel imports.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6010001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/task.yaml index d7f58fb08c..47092a00f2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-rr-unmapped-drive/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/README.md new file mode 100644 index 0000000000..f67ad49938 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/README.md @@ -0,0 +1,59 @@ +# Excel Application Scope — COM Add-in Interface Clash + +This scenario reproduces a Classic `Excel Application Scope` failure +where a COM add-in loaded into the Excel instance corrupts the interop +interface negotiation. The job ends with: + +``` +System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'. +``` + +(QueryInterface for IID `{000208D5-...}` → `0x80004002 E_NOINTERFACE`) + +## What this scenario uncovers + +**Root Cause:** The scope instantiates `Excel.Application`, but the +`UiPath.Integration.ExcelAddin` (the Studio design-time Excel Add-in, +`LoadBehavior=3`, registered on this runtime host where it shouldn't be) +— alongside a third-party `AcmeBudgetTools.Connect` add-in — corrupts +the QueryInterface for `Microsoft.Office.Interop.Excel._Application`, +returning `E_NOINTERFACE`. The `System.__ComObject` can't be cast to the +interop interface. `excel.exe /safe` (add-ins disabled) succeeds, +confirming an add-in is the culprit. + +The fix is to disable the offending COM add-in (File → Options → +Add-ins → COM Add-ins, or `LoadBehavior=0` in the registry) and confirm +with `excel.exe /safe`. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-scope-failures.md` +(branch 3 — COM add-in interface clash). + +## The trap + +The error mentions COM and Excel, which can tempt an agent toward +"Excel isn't installed / repair Excel." But Excel IS installed and +launches, and the inner HRESULT is `E_NOINTERFACE` (an interface-cast +fault), not `REGDB_E_CLASSNOTREG` / `TYPE_E_LIBNOTREGISTERED`. The +`/safe` success is the tell: the COM stack is healthy without add-ins. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelReconExport` — single Classic `` wrapping a `Read Range` | + +Expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (`InvalidCastException`, +`E_NOINTERFACE`) → `jobs logs` (loaded COM add-ins + `/safe` succeeds) → +workflow source (single Classic scope) → conclude branch 3. + +> **Note on fixtures.** Synthetic. The IID, add-in names, and +> `LoadBehavior` values are placeholders. The test grades whether the +> agent identifies the COM add-in clash (NOT not-installed, NOT a lock) +> and recommends disabling the offending add-in (UI or `LoadBehavior=0`) +> with `/safe` confirmation. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/289d2a67b7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/289d2a67b7.json deleted file mode 100644 index 7300310a56..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/289d2a67b7.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-30T07:30:01.205Z"}, - {"Key": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-30T07:30:01.500Z"}, - {"Key": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-30T07:30:02.640Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/429111d7a7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/429111d7a7.json deleted file mode 100644 index 4d0597f56e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/429111d7a7.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T07:30:01.500Z", - "EndTime": "2026-05-30T07:30:02.640Z", - "State": "Faulted", - "ReleaseName": "ExcelReconExport", - "Source": "Schedule", - "Reference": "DailyRecon0730", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0066666-3333-4444-5555-666677778888", - "Info": "System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/6194634ea7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/6194634ea7.json deleted file mode 100644 index cbb7328bec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/6194634ea7.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0066666-3333-4444-5555-666677778888", - "DisplayName": "ReconReports", - "FullyQualifiedName": "ReconReports", - "Description": "Folder for the daily reconciliation export automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6110066, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/80ce0cb156.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/80ce0cb156.json deleted file mode 100644 index 50ed50f086..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/80ce0cb156.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T07:30:01.500Z", - "EndTime": "2026-05-30T07:30:02.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa66666-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{000208D5-0000-0000-C000-000000000046}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).\r\n at UiPath.Excel.Activities.ExcelApplicationScope.AcquireExcelComServer()\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)", - "CreationTime": "2026-05-30T07:30:01.205Z", - "ReleaseName": "ExcelReconExport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681166, - "OrganizationUnitId": 6110066, - "FolderKey": "f0066666-3333-4444-5555-666677778888", - "Reference": "DailyRecon0730", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T07:30:02.640Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "ApplicationException", - "ActivityName": "ExcelApplicationScope_1", - "ActivityDisplayName": "Excel Application Scope: recon-daily.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681199066 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/README.md deleted file mode 100644 index 4cd54f46d5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# Excel Application Scope — COM Add-in Interface Clash - -This scenario reproduces a Classic `Excel Application Scope` failure -where a COM add-in loaded into the Excel instance corrupts the interop -interface negotiation. The job ends with: - -``` -System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'. -``` - -(QueryInterface for IID `{000208D5-...}` → `0x80004002 E_NOINTERFACE`) - -## What this scenario uncovers - -**Root Cause:** The scope instantiates `Excel.Application`, but the -`UiPath.Integration.ExcelAddin` (the Studio design-time Excel Add-in, -`LoadBehavior=3`, registered on this runtime host where it shouldn't be) -— alongside a third-party `AcmeBudgetTools.Connect` add-in — corrupts -the QueryInterface for `Microsoft.Office.Interop.Excel._Application`, -returning `E_NOINTERFACE`. The `System.__ComObject` can't be cast to the -interop interface. `excel.exe /safe` (add-ins disabled) succeeds, -confirming an add-in is the culprit. - -The fix is to disable the offending COM add-in (File → Options → -Add-ins → COM Add-ins, or `LoadBehavior=0` in the registry) and confirm -with `excel.exe /safe`. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-scope-failures.md` -(branch 3 — COM add-in interface clash). - -## The trap - -The error mentions COM and Excel, which can tempt an agent toward -"Excel isn't installed / repair Excel." But Excel IS installed and -launches, and the inner HRESULT is `E_NOINTERFACE` (an interface-cast -fault), not `REGDB_E_CLASSNOTREG` / `TYPE_E_LIBNOTREGISTERED`. The -`/safe` success is the tell: the COM stack is healthy without add-ins. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelReconExport` — single Classic `` wrapping a `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `jobs get` carries the `InvalidCastException` / `E_NOINTERFACE`, and `jobs logs` enumerates the loaded COM add-ins (incl. `UiPath.Integration.ExcelAddin`) and notes `excel.exe /safe` succeeds | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -Expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (`InvalidCastException`, -`E_NOINTERFACE`) → `jobs logs` (loaded COM add-ins + `/safe` succeeds) → -workflow source (single Classic scope) → conclude branch 3. - -> **Note on fixtures.** Synthetic. The IID, add-in names, and -> `LoadBehavior` values are placeholders. The test grades whether the -> agent identifies the COM add-in clash (NOT not-installed, NOT a lock) -> and recommends disabling the offending add-in (UI or `LoadBehavior=0`) -> with `/safe` confirmation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/d71286fc73.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/d71286fc73.json deleted file mode 100644 index 8adbd8fe85..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/d71286fc73.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa6666cc-001", - "JobKey": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T07:30:01.520Z", - "Message": "Execution started.", - "ProcessName": "ExcelReconExport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa6666cc-002", - "JobKey": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T07:30:01.640Z", - "Message": "[ExcelReconExport] Opening C:\\Robot\\Data\\recon-daily.xlsx to export the reconciliation summary", - "ProcessName": "ExcelReconExport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa6666cc-003", - "JobKey": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T07:30:01.820Z", - "Message": "Excel Application Scope: recon-daily.xlsx — surface: Classic ExcelApplicationScope (COM-only). Attempting to acquire Excel.Application COM server.", - "ProcessName": "ExcelReconExport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa6666cc-004", - "JobKey": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T07:30:02.300Z", - "Message": "Excel Application Scope: recon-daily.xlsx — Excel.Application object instantiated, but QueryInterface for Microsoft.Office.Interop.Excel._Application (IID {000208D5-0000-0000-C000-000000000046}) returned E_NOINTERFACE. Host: MOCK-HOST. Excel install state: INSTALLED (Microsoft 365 Apps, launches interactively). COM add-ins auto-loaded into this Excel instance at startup: 'UiPath.Integration.ExcelAddin' (LoadBehavior=3, HKLM — the Studio design-time Excel Add-in, present on this runtime host) and 'AcmeBudgetTools.Connect' (LoadBehavior=3). Diagnostic: launching 'excel.exe /safe' (all add-ins disabled) on the host succeeds and a manual COM bind to _Application works.", - "ProcessName": "ExcelReconExport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa6666cc-005", - "JobKey": "aa666666-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-30T07:30:02.620Z", - "Message": "Excel Application Scope: recon-daily.xlsx — System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'. QueryInterface for IID {000208D5-0000-0000-C000-000000000046} failed: 0x80004002 E_NOINTERFACE (No such interface supported).", - "ProcessName": "ExcelReconExport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/manifest.json deleted file mode 100644 index 39fce92d3d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Application Scope reconciliation-export scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "6194634ea7.json"}, - {"match": "or folders list --output json", "file": "6194634ea7.json"}, - {"match": "or jobs list --folder-key \"f0066666-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "429111d7a7.json"}, - {"match": "or jobs list --folder-key f0066666-3333-4444-5555-666677778888 --state Faulted --output json", "file": "429111d7a7.json"}, - {"match": "or jobs get \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "80ce0cb156.json"}, - {"match": "or jobs get aa666666-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "80ce0cb156.json"}, - {"match": "or jobs history \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "289d2a67b7.json"}, - {"match": "or jobs history aa666666-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "289d2a67b7.json"}, - {"match": "or jobs logs \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "d71286fc73.json"}, - {"match": "or jobs logs aa666666-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "d71286fc73.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/uip-fixture.json new file mode 100644 index 0000000000..b778e6ff92 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0066666-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T07:30:01.500Z\",\r\n \"EndTime\": \"2026-05-30T07:30:02.640Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelReconExport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailyRecon0730\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0066666-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa666666-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T07:30:01.500Z\",\r\n \"EndTime\": \"2026-05-30T07:30:02.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa66666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{000208D5-0000-0000-C000-000000000046}' failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.AcquireExcelComServer()\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\",\r\n \"CreationTime\": \"2026-05-30T07:30:01.205Z\",\r\n \"ReleaseName\": \"ExcelReconExport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681166,\r\n \"OrganizationUnitId\": 6110066,\r\n \"FolderKey\": \"f0066666-3333-4444-5555-666677778888\",\r\n \"Reference\": \"DailyRecon0730\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T07:30:02.640Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"ApplicationException\",\r\n \"ActivityName\": \"ExcelApplicationScope_1\",\r\n \"ActivityDisplayName\": \"Excel Application Scope: recon-daily.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681199066\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa666666-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-30T07:30:01.205Z\"},\r\n {\"Key\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-30T07:30:01.500Z\"},\r\n {\"Key\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-30T07:30:02.640Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa666666-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa6666cc-001\",\r\n \"JobKey\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T07:30:01.520Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelReconExport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa6666cc-002\",\r\n \"JobKey\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T07:30:01.640Z\",\r\n \"Message\": \"[ExcelReconExport] Opening C:\\\\Robot\\\\Data\\\\recon-daily.xlsx to export the reconciliation summary\",\r\n \"ProcessName\": \"ExcelReconExport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa6666cc-003\",\r\n \"JobKey\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T07:30:01.820Z\",\r\n \"Message\": \"Excel Application Scope: recon-daily.xlsx — surface: Classic ExcelApplicationScope (COM-only). Attempting to acquire Excel.Application COM server.\",\r\n \"ProcessName\": \"ExcelReconExport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa6666cc-004\",\r\n \"JobKey\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T07:30:02.300Z\",\r\n \"Message\": \"Excel Application Scope: recon-daily.xlsx — Excel.Application object instantiated, but QueryInterface for Microsoft.Office.Interop.Excel._Application (IID {000208D5-0000-0000-C000-000000000046}) returned E_NOINTERFACE. Host: MOCK-HOST. Excel install state: INSTALLED (Microsoft 365 Apps, launches interactively). COM add-ins auto-loaded into this Excel instance at startup: 'UiPath.Integration.ExcelAddin' (LoadBehavior=3, HKLM — the Studio design-time Excel Add-in, present on this runtime host) and 'AcmeBudgetTools.Connect' (LoadBehavior=3). Diagnostic: launching 'excel.exe /safe' (all add-ins disabled) on the host succeeds and a manual COM bind to _Application works.\",\r\n \"ProcessName\": \"ExcelReconExport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa6666cc-005\",\r\n \"JobKey\": \"aa666666-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-30T07:30:02.620Z\",\r\n \"Message\": \"Excel Application Scope: recon-daily.xlsx — System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Excel._Application'. QueryInterface for IID {000208D5-0000-0000-C000-000000000046} failed: 0x80004002 E_NOINTERFACE (No such interface supported).\",\r\n \"ProcessName\": \"ExcelReconExport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0066666-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"ReconReports\",\r\n \"FullyQualifiedName\": \"ReconReports\",\r\n \"Description\": \"Folder for the daily reconciliation export automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6110066,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/task.yaml index 785c3a9c59..267387d1c9 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-addin-clash/task.yaml @@ -28,21 +28,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/README.md new file mode 100644 index 0000000000..fdb1c77a6a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/README.md @@ -0,0 +1,67 @@ +# Excel Application Scope — COM Registration Broken After Office Upgrade + +This scenario reproduces a Classic `Excel Application Scope` failure on +a Robot host where **Excel IS installed** but its `Excel.Application` +COM / type-library registration was corrupted by a recent in-place +Microsoft 365 upgrade. The job ends with: + +``` +UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. +``` + +(with inner `COMException 0x8002801D TYPE_E_LIBNOTREGISTERED`) + +## What this scenario uncovers + +**Root Cause:** Excel is present (Microsoft 365 Apps build path logged, +launches interactively), but an in-place upgrade from Office 2019 left +a stale `TypeLib` registration for the Excel Object Library. The COM +class factory creates the `Excel.Application` object, then interop +type-library resolution fails with `TYPE_E_LIBNOTREGISTERED`. The +Classic scope rewraps it as the generic "Make sure Excel is installed." + +The fix is to repair the COM / type-library registration (Office +Online Repair, UiPath Repair Tool, or targeted `TypeLib` cleanup) — +**not** to install Excel, which is already installed. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-scope-failures.md` +(branch 1 — COM registration corruption). + +## Why this is the hard contrast case + +This scenario is the deliberate counterpart to +`excel-card-excel-not-installed`. Both throw the identical +`BusinessException: Error opening workbook. Make sure Excel is +installed.` The agent must NOT reflexively conclude "Excel isn't +installed — install it." The distinguishing evidence is: + +| Signal | Not-installed scenario | This scenario (registration broken) | +|---|---|---| +| Inner HRESULT | `0x80040154 REGDB_E_CLASSNOTREG` | `0x8002801D TYPE_E_LIBNOTREGISTERED` | +| Host Excel state (logs) | NOT INSTALLED (no registry, no excel.exe) | INSTALLED — build path given, launches | +| Trigger | new deployment to a headless host | recent in-place Office upgrade | +| Fix | install Excel OR migrate off COM | repair COM/TypeLib registration | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelLedgerReport` — single Classic `` wrapping a `Read Range` | + +Expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException, inner +`TYPE_E_LIBNOTREGISTERED`) → `jobs logs` (INSTALLED state + upgrade + +stale TypeLib) → workflow source (single Classic scope) → conclude +branch 1 (registration corruption). + +> **Note on fixtures.** Synthetic. The exact build numbers, TypeLib +> GUID handling, and host-detection wording are placeholders. The test +> grades whether the agent recognizes Excel-is-installed-but-COM- +> registration-is-broken (NOT not-installed) and recommends a repair +> (Online Repair / UiPath Repair Tool / TypeLib cleanup) rather than +> "install Excel". + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/45e7959dc5.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/45e7959dc5.json deleted file mode 100644 index adc9aa5c90..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/45e7959dc5.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa4444cc-001", - "JobKey": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T09:15:02.120Z", - "Message": "Execution started.", - "ProcessName": "ExcelLedgerReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa4444cc-002", - "JobKey": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T09:15:02.240Z", - "Message": "[ExcelLedgerReport] Opening C:\\Robot\\Data\\ledger-2026-05.xlsm to recalc the monthly ledger", - "ProcessName": "ExcelLedgerReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa4444cc-003", - "JobKey": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:15:02.460Z", - "Message": "Excel Application Scope: ledger-2026-05.xlsm — surface: Classic ExcelApplicationScope (COM-only). Attempting to acquire Excel.Application COM server.", - "ProcessName": "ExcelLedgerReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa4444cc-004", - "JobKey": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T09:15:02.980Z", - "Message": "Excel Application Scope: ledger-2026-05.xlsm — Excel.Application class factory created, but Microsoft.Office.Interop.Excel type-library resolution FAILED. Host: MOCK-HOST. Robot user: UIPATH\\AUTOMATION1. Excel install state: INSTALLED — Microsoft 365 Apps build 16.0.17928.20114 at C:\\Program Files\\Microsoft Office\\root\\Office16\\EXCEL.EXE (launches interactively). Office was upgraded in place to Microsoft 365 Apps on 2026-05-28 (previous build: Office 2019 Standard 16.0.10396). Stale TypeLib registration detected for {00020813-0000-0000-C000-000000000046} (Microsoft Excel Object Library) pointing at the removed 2019 build.", - "ProcessName": "ExcelLedgerReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa4444cc-005", - "JobKey": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-30T09:15:03.390Z", - "Message": "Excel Application Scope: ledger-2026-05.xlsm — UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. Inner: COMException 0x8002801D TYPE_E_LIBNOTREGISTERED (Library not registered).", - "ProcessName": "ExcelLedgerReport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/5280357351.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/5280357351.json deleted file mode 100644 index c7cd06f3a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/5280357351.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T09:15:02.110Z", - "EndTime": "2026-05-30T09:15:03.402Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa44444-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.\r\n at UiPath.Excel.Activities.ExcelApplicationScope.AcquireExcelComServer()\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\r\nInner: System.Runtime.InteropServices.COMException (0x8002801D): Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))", - "CreationTime": "2026-05-30T09:15:01.900Z", - "ReleaseName": "ExcelLedgerReport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681144, - "OrganizationUnitId": 6110044, - "FolderKey": "f00eeeee-3333-4444-5555-666677778888", - "Reference": "MonthlyLedger", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T09:15:03.402Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelApplicationScope_1", - "ActivityDisplayName": "Excel Application Scope: ledger-2026-05.xlsm", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681199044 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/6097e75792.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/6097e75792.json deleted file mode 100644 index 0bcdfbbda4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/6097e75792.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T09:15:02.110Z", - "EndTime": "2026-05-30T09:15:03.402Z", - "State": "Faulted", - "ReleaseName": "ExcelLedgerReport", - "Source": "Schedule", - "Reference": "MonthlyLedger", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00eeeee-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/8140b32704.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/8140b32704.json deleted file mode 100644 index 504a80defa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/8140b32704.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f00eeeee-3333-4444-5555-666677778888", - "DisplayName": "LedgerReports", - "FullyQualifiedName": "LedgerReports", - "Description": "Folder for the monthly ledger recalculation automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6110044, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/README.md deleted file mode 100644 index 4324c6f355..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/README.md +++ /dev/null @@ -1,66 +0,0 @@ -# Excel Application Scope — COM Registration Broken After Office Upgrade - -This scenario reproduces a Classic `Excel Application Scope` failure on -a Robot host where **Excel IS installed** but its `Excel.Application` -COM / type-library registration was corrupted by a recent in-place -Microsoft 365 upgrade. The job ends with: - -``` -UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. -``` - -(with inner `COMException 0x8002801D TYPE_E_LIBNOTREGISTERED`) - -## What this scenario uncovers - -**Root Cause:** Excel is present (Microsoft 365 Apps build path logged, -launches interactively), but an in-place upgrade from Office 2019 left -a stale `TypeLib` registration for the Excel Object Library. The COM -class factory creates the `Excel.Application` object, then interop -type-library resolution fails with `TYPE_E_LIBNOTREGISTERED`. The -Classic scope rewraps it as the generic "Make sure Excel is installed." - -The fix is to repair the COM / type-library registration (Office -Online Repair, UiPath Repair Tool, or targeted `TypeLib` cleanup) — -**not** to install Excel, which is already installed. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-scope-failures.md` -(branch 1 — COM registration corruption). - -## Why this is the hard contrast case - -This scenario is the deliberate counterpart to -`excel-card-excel-not-installed`. Both throw the identical -`BusinessException: Error opening workbook. Make sure Excel is -installed.` The agent must NOT reflexively conclude "Excel isn't -installed — install it." The distinguishing evidence is: - -| Signal | Not-installed scenario | This scenario (registration broken) | -|---|---|---| -| Inner HRESULT | `0x80040154 REGDB_E_CLASSNOTREG` | `0x8002801D TYPE_E_LIBNOTREGISTERED` | -| Host Excel state (logs) | NOT INSTALLED (no registry, no excel.exe) | INSTALLED — build path given, launches | -| Trigger | new deployment to a headless host | recent in-place Office upgrade | -| Fix | install Excel OR migrate off COM | repair COM/TypeLib registration | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelLedgerReport` — single Classic `` wrapping a `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries pin (a) Classic surface, (b) Excel INSTALLED with a concrete build path + recent Office upgrade, (c) inner `TYPE_E_LIBNOTREGISTERED` + stale TypeLib | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -Expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException, inner -`TYPE_E_LIBNOTREGISTERED`) → `jobs logs` (INSTALLED state + upgrade + -stale TypeLib) → workflow source (single Classic scope) → conclude -branch 1 (registration corruption). - -> **Note on fixtures.** Synthetic. The exact build numbers, TypeLib -> GUID handling, and host-detection wording are placeholders. The test -> grades whether the agent recognizes Excel-is-installed-but-COM- -> registration-is-broken (NOT not-installed) and recommends a repair -> (Online Repair / UiPath Repair Tool / TypeLib cleanup) rather than -> "install Excel". diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/e66a824b5d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/e66a824b5d.json deleted file mode 100644 index 04b15a556b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/e66a824b5d.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-30T09:15:01.900Z"}, - {"Key": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-30T09:15:02.110Z"}, - {"Key": "aa444444-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-30T09:15:03.402Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/manifest.json deleted file mode 100644 index 0822c43b5f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Application Scope ledger-report scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "8140b32704.json"}, - {"match": "or folders list --output json", "file": "8140b32704.json"}, - {"match": "or jobs list --folder-key \"f00eeeee-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "6097e75792.json"}, - {"match": "or jobs list --folder-key f00eeeee-3333-4444-5555-666677778888 --state Faulted --output json", "file": "6097e75792.json"}, - {"match": "or jobs get \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "5280357351.json"}, - {"match": "or jobs get aa444444-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "5280357351.json"}, - {"match": "or jobs history \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "e66a824b5d.json"}, - {"match": "or jobs history aa444444-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "e66a824b5d.json"}, - {"match": "or jobs logs \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "45e7959dc5.json"}, - {"match": "or jobs logs aa444444-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "45e7959dc5.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/uip-fixture.json new file mode 100644 index 0000000000..571916bf18 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f00eeeee-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T09:15:02.110Z\",\r\n \"EndTime\": \"2026-05-30T09:15:03.402Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelLedgerReport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"MonthlyLedger\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00eeeee-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa444444-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T09:15:02.110Z\",\r\n \"EndTime\": \"2026-05-30T09:15:03.402Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa44444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed.\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.AcquireExcelComServer()\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\\r\\nInner: System.Runtime.InteropServices.COMException (0x8002801D): Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))\",\r\n \"CreationTime\": \"2026-05-30T09:15:01.900Z\",\r\n \"ReleaseName\": \"ExcelLedgerReport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681144,\r\n \"OrganizationUnitId\": 6110044,\r\n \"FolderKey\": \"f00eeeee-3333-4444-5555-666677778888\",\r\n \"Reference\": \"MonthlyLedger\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T09:15:03.402Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelApplicationScope_1\",\r\n \"ActivityDisplayName\": \"Excel Application Scope: ledger-2026-05.xlsm\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681199044\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa444444-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-30T09:15:01.900Z\"},\r\n {\"Key\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-30T09:15:02.110Z\"},\r\n {\"Key\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-30T09:15:03.402Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa444444-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa4444cc-001\",\r\n \"JobKey\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T09:15:02.120Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelLedgerReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa4444cc-002\",\r\n \"JobKey\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T09:15:02.240Z\",\r\n \"Message\": \"[ExcelLedgerReport] Opening C:\\\\Robot\\\\Data\\\\ledger-2026-05.xlsm to recalc the monthly ledger\",\r\n \"ProcessName\": \"ExcelLedgerReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa4444cc-003\",\r\n \"JobKey\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:15:02.460Z\",\r\n \"Message\": \"Excel Application Scope: ledger-2026-05.xlsm — surface: Classic ExcelApplicationScope (COM-only). Attempting to acquire Excel.Application COM server.\",\r\n \"ProcessName\": \"ExcelLedgerReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa4444cc-004\",\r\n \"JobKey\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T09:15:02.980Z\",\r\n \"Message\": \"Excel Application Scope: ledger-2026-05.xlsm — Excel.Application class factory created, but Microsoft.Office.Interop.Excel type-library resolution FAILED. Host: MOCK-HOST. Robot user: UIPATH\\\\AUTOMATION1. Excel install state: INSTALLED — Microsoft 365 Apps build 16.0.17928.20114 at C:\\\\Program Files\\\\Microsoft Office\\\\root\\\\Office16\\\\EXCEL.EXE (launches interactively). Office was upgraded in place to Microsoft 365 Apps on 2026-05-28 (previous build: Office 2019 Standard 16.0.10396). Stale TypeLib registration detected for {00020813-0000-0000-C000-000000000046} (Microsoft Excel Object Library) pointing at the removed 2019 build.\",\r\n \"ProcessName\": \"ExcelLedgerReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa4444cc-005\",\r\n \"JobKey\": \"aa444444-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-30T09:15:03.390Z\",\r\n \"Message\": \"Excel Application Scope: ledger-2026-05.xlsm — UiPath.Excel.BusinessException: Error opening workbook. Make sure Excel is installed. Inner: COMException 0x8002801D TYPE_E_LIBNOTREGISTERED (Library not registered).\",\r\n \"ProcessName\": \"ExcelLedgerReport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f00eeeee-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"LedgerReports\",\r\n \"FullyQualifiedName\": \"LedgerReports\",\r\n \"Description\": \"Folder for the monthly ledger recalculation automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6110044,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/task.yaml index f340165d20..d6616d3fdc 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-com-registration-broken/task.yaml @@ -28,21 +28,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/README.md new file mode 100644 index 0000000000..7362c31051 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/README.md @@ -0,0 +1,64 @@ +# Excel Application Scope — Workbook Locked by an Orphan EXCEL.EXE + +This scenario reproduces a Classic `Excel Application Scope` failure +where the target workbook is locked by an **orphan `EXCEL.EXE`** that a +prior crashed job left running. The job ends with: + +``` +UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected. +``` + +(with inner `COMException 0x800A03EC` — "Microsoft Excel cannot access the file … in use by another program") + +## What this scenario uncovers + +**Root Cause:** A prior `ExcelOrderImport` job (`gg5550000-...`) faulted +mid-scope on 2026-05-29 when its Robot session was force-terminated, +leaving an `EXCEL.EXE` (PID 7312, session 0, no window) still holding +the workbook handle. The current job's Classic scope can't open the +file exclusively, so it throws "Failed opening the Excel file." + +The fix is to clear the orphan (`Stop-Process` / Task Manager; or a +guarded `Kill Process EXCEL` at workflow start on a dedicated +unattended host) AND stop creating orphans (clean scope close, no +force-kill mid-scope, `Excel Process Scope` for multi-scope flows) — +**not** to treat the file as corrupt or password protected. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-scope-failures.md` +(branch 2 — workbook held by another EXCEL.EXE). + +## The trap + +The exception message lists THREE possible reasons — "file is corrupt, +already used by another process or password protected." Only the +middle one applies. The agent must use the log evidence (which +explicitly rules out corruption and password protection and names the +orphan process) rather than chasing the corrupt/password red herrings. + +It must also scope its `Kill Process EXCEL` recommendation correctly: +valid recovery on a dedicated unattended host, dangerous on a +shared/attended one. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelOrderImport` — single Classic `` wrapping a `Read Range` | + +Expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` (sees this run + the prior crashed run) → +`jobs get` (Failed opening / inner `0x800A03EC` in-use) → `jobs logs` +(orphan `EXCEL.EXE` PID 7312 + originating crashed job; corruption and +password ruled out) → workflow source (single Classic scope) → +conclude branch 2. + +> **Note on fixtures.** Synthetic. The PID, session id, and +> orphan-detection wording are placeholders. The test grades whether +> the agent identifies the orphan-process lock (NOT corruption / +> password), recommends clearing it + fixing the orphan source, and +> scopes the `Kill Process EXCEL` advice to dedicated unattended hosts. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/61a7fc5eb1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/61a7fc5eb1.json deleted file mode 100644 index ec682a4b45..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/61a7fc5eb1.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T06:00:02.300Z", - "EndTime": "2026-05-30T06:00:09.870Z", - "State": "Faulted", - "ReleaseName": "ExcelOrderImport", - "Source": "Schedule", - "Reference": "InboundOrders0600", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00fffff-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected.", - "ErrorCode": "Activity" - }, - { - "Key": "ga5550000-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-29T22:03:55.100Z", - "EndTime": "2026-05-29T22:04:11.540Z", - "State": "Faulted", - "ReleaseName": "ExcelOrderImport", - "Source": "Schedule", - "Reference": "InboundOrders2200", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f00fffff-3333-4444-5555-666677778888", - "Info": "System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Robot session force-terminated mid-scope.)", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/737fe05651.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/737fe05651.json deleted file mode 100644 index 99487f8a60..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/737fe05651.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-30T06:00:02.300Z", - "EndTime": "2026-05-30T06:00:09.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa55555-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected.\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OpenWorkbook(String path)\r\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\r\nInner: System.Runtime.InteropServices.COMException (0x800A03EC): Microsoft Excel cannot access the file 'C:\\Robot\\Data\\orders-inbound.xlsx'. There are several possible reasons: the file name or path does not exist; the file is being used by another program; the workbook you are trying to save has the same name as a currently open workbook.", - "CreationTime": "2026-05-30T06:00:02.005Z", - "ReleaseName": "ExcelOrderImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5681155, - "OrganizationUnitId": 6110055, - "FolderKey": "f00fffff-3333-4444-5555-666677778888", - "Reference": "InboundOrders0600", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-30T06:00:09.870Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "ExcelApplicationScope_1", - "ActivityDisplayName": "Excel Application Scope: orders-inbound.xlsx", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 681199055 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/README.md deleted file mode 100644 index 75d9b27d4c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# Excel Application Scope — Workbook Locked by an Orphan EXCEL.EXE - -This scenario reproduces a Classic `Excel Application Scope` failure -where the target workbook is locked by an **orphan `EXCEL.EXE`** that a -prior crashed job left running. The job ends with: - -``` -UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected. -``` - -(with inner `COMException 0x800A03EC` — "Microsoft Excel cannot access the file … in use by another program") - -## What this scenario uncovers - -**Root Cause:** A prior `ExcelOrderImport` job (`gg5550000-...`) faulted -mid-scope on 2026-05-29 when its Robot session was force-terminated, -leaving an `EXCEL.EXE` (PID 7312, session 0, no window) still holding -the workbook handle. The current job's Classic scope can't open the -file exclusively, so it throws "Failed opening the Excel file." - -The fix is to clear the orphan (`Stop-Process` / Task Manager; or a -guarded `Kill Process EXCEL` at workflow start on a dedicated -unattended host) AND stop creating orphans (clean scope close, no -force-kill mid-scope, `Excel Process Scope` for multi-scope flows) — -**not** to treat the file as corrupt or password protected. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/excel-application-scope-failures.md` -(branch 2 — workbook held by another EXCEL.EXE). - -## The trap - -The exception message lists THREE possible reasons — "file is corrupt, -already used by another process or password protected." Only the -middle one applies. The agent must use the log evidence (which -explicitly rules out corruption and password protection and names the -orphan process) rather than chasing the corrupt/password red herrings. - -It must also scope its `Kill Process EXCEL` recommendation correctly: -valid recovery on a dedicated unattended host, dangerous on a -shared/attended one. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelOrderImport` — single Classic `` wrapping a `Read Range` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `jobs list` includes the prior force-terminated run, and `jobs logs` names the orphan `EXCEL.EXE` lock holder + its origin and rules out corruption/password | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -Expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` (sees this run + the prior crashed run) → -`jobs get` (Failed opening / inner `0x800A03EC` in-use) → `jobs logs` -(orphan `EXCEL.EXE` PID 7312 + originating crashed job; corruption and -password ruled out) → workflow source (single Classic scope) → -conclude branch 2. - -> **Note on fixtures.** Synthetic. The PID, session id, and -> orphan-detection wording are placeholders. The test grades whether -> the agent identifies the orphan-process lock (NOT corruption / -> password), recommends clearing it + fixing the orphan source, and -> scopes the `Kill Process EXCEL` advice to dedicated unattended hosts. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/a16af76c21.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/a16af76c21.json deleted file mode 100644 index cca2a8dc2f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/a16af76c21.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa5555cc-001", - "JobKey": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T06:00:02.320Z", - "Message": "Execution started.", - "ProcessName": "ExcelOrderImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa5555cc-002", - "JobKey": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-30T06:00:02.480Z", - "Message": "[ExcelOrderImport] Opening C:\\Robot\\Data\\orders-inbound.xlsx to import inbound orders", - "ProcessName": "ExcelOrderImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa5555cc-003", - "JobKey": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T06:00:02.700Z", - "Message": "Excel Application Scope: orders-inbound.xlsx — surface: Classic ExcelApplicationScope (COM-only). Acquiring workbook via Excel COM.", - "ProcessName": "ExcelOrderImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa5555cc-004", - "JobKey": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-30T06:00:08.900Z", - "Message": "Excel Application Scope: orders-inbound.xlsx — workbook open failed: target file is locked. Host: MOCK-HOST. Lock holder: EXCEL.EXE PID 7312, session 0 (Robot session UIPATH\\AUTOMATION1), NO main window — orphan instance (not a logged-in user's Excel). Orphan originated from job ga5550000-cccc-dddd-eeee-ffffaaaabbbb (ExcelOrderImport, Faulted 2026-05-29T22:04:11 mid-scope; the Robot session was force-terminated and EXCEL.EXE was left running, still holding the workbook handle). The file itself is well-formed (opens read-only from a copy) and is not password protected.", - "ProcessName": "ExcelOrderImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa5555cc-005", - "JobKey": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-30T06:00:09.850Z", - "Message": "Excel Application Scope: orders-inbound.xlsx — UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected. Inner: COMException 0x800A03EC (Microsoft Excel cannot access the file; in use by another process).", - "ProcessName": "ExcelOrderImport", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/d7f1661bad.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/d7f1661bad.json deleted file mode 100644 index 6d9d31f6ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/d7f1661bad.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f00fffff-3333-4444-5555-666677778888", - "DisplayName": "OrderImports", - "FullyQualifiedName": "OrderImports", - "Description": "Folder for the inbound order-import automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6110055, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/e8a675aab7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/e8a675aab7.json deleted file mode 100644 index 18b7422370..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/e8a675aab7.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-30T06:00:02.005Z"}, - {"Key": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-30T06:00:02.300Z"}, - {"Key": "aa555555-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-30T06:00:09.870Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/manifest.json deleted file mode 100644 index 83a3b97ee5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Application Scope order-import scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "d7f1661bad.json"}, - {"match": "or folders list --output json", "file": "d7f1661bad.json"}, - {"match": "or jobs list --folder-key \"f00fffff-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "61a7fc5eb1.json"}, - {"match": "or jobs list --folder-key f00fffff-3333-4444-5555-666677778888 --state Faulted --output json", "file": "61a7fc5eb1.json"}, - {"match": "or jobs get \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "737fe05651.json"}, - {"match": "or jobs get aa555555-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "737fe05651.json"}, - {"match": "or jobs history \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "e8a675aab7.json"}, - {"match": "or jobs history aa555555-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "e8a675aab7.json"}, - {"match": "or jobs logs \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "a16af76c21.json"}, - {"match": "or jobs logs aa555555-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "a16af76c21.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/uip-fixture.json new file mode 100644 index 0000000000..9e2c44f14d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f00fffff-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T06:00:02.300Z\",\r\n \"EndTime\": \"2026-05-30T06:00:09.870Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelOrderImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"InboundOrders0600\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00fffff-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected.\",\r\n \"ErrorCode\": \"Activity\"\r\n },\r\n {\r\n \"Key\": \"ga5550000-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-29T22:03:55.100Z\",\r\n \"EndTime\": \"2026-05-29T22:04:11.540Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelOrderImport\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"InboundOrders2200\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f00fffff-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Robot session force-terminated mid-scope.)\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aa555555-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-30T06:00:02.300Z\",\r\n \"EndTime\": \"2026-05-30T06:00:09.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa55555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected.\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OpenWorkbook(String path)\\r\\n at UiPath.Excel.Activities.ExcelApplicationScope.OnExecute(NativeActivityContext context)\\r\\nInner: System.Runtime.InteropServices.COMException (0x800A03EC): Microsoft Excel cannot access the file 'C:\\\\Robot\\\\Data\\\\orders-inbound.xlsx'. There are several possible reasons: the file name or path does not exist; the file is being used by another program; the workbook you are trying to save has the same name as a currently open workbook.\",\r\n \"CreationTime\": \"2026-05-30T06:00:02.005Z\",\r\n \"ReleaseName\": \"ExcelOrderImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5681155,\r\n \"OrganizationUnitId\": 6110055,\r\n \"FolderKey\": \"f00fffff-3333-4444-5555-666677778888\",\r\n \"Reference\": \"InboundOrders0600\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-30T06:00:09.870Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"ExcelApplicationScope_1\",\r\n \"ActivityDisplayName\": \"Excel Application Scope: orders-inbound.xlsx\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 681199055\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "aa555555-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-30T06:00:02.005Z\"},\r\n {\"Key\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-30T06:00:02.300Z\"},\r\n {\"Key\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-30T06:00:09.870Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aa555555-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa5555cc-001\",\r\n \"JobKey\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T06:00:02.320Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelOrderImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa5555cc-002\",\r\n \"JobKey\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-30T06:00:02.480Z\",\r\n \"Message\": \"[ExcelOrderImport] Opening C:\\\\Robot\\\\Data\\\\orders-inbound.xlsx to import inbound orders\",\r\n \"ProcessName\": \"ExcelOrderImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa5555cc-003\",\r\n \"JobKey\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T06:00:02.700Z\",\r\n \"Message\": \"Excel Application Scope: orders-inbound.xlsx — surface: Classic ExcelApplicationScope (COM-only). Acquiring workbook via Excel COM.\",\r\n \"ProcessName\": \"ExcelOrderImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa5555cc-004\",\r\n \"JobKey\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-30T06:00:08.900Z\",\r\n \"Message\": \"Excel Application Scope: orders-inbound.xlsx — workbook open failed: target file is locked. Host: MOCK-HOST. Lock holder: EXCEL.EXE PID 7312, session 0 (Robot session UIPATH\\\\AUTOMATION1), NO main window — orphan instance (not a logged-in user's Excel). Orphan originated from job ga5550000-cccc-dddd-eeee-ffffaaaabbbb (ExcelOrderImport, Faulted 2026-05-29T22:04:11 mid-scope; the Robot session was force-terminated and EXCEL.EXE was left running, still holding the workbook handle). The file itself is well-formed (opens read-only from a copy) and is not password protected.\",\r\n \"ProcessName\": \"ExcelOrderImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa5555cc-005\",\r\n \"JobKey\": \"aa555555-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-30T06:00:09.850Z\",\r\n \"Message\": \"Excel Application Scope: orders-inbound.xlsx — UiPath.Excel.BusinessException: Failed opening the Excel file. Possible reasons: file is corrupt, already used by another process or password protected. Inner: COMException 0x800A03EC (Microsoft Excel cannot access the file; in use by another process).\",\r\n \"ProcessName\": \"ExcelOrderImport\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f00fffff-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"OrderImports\",\r\n \"FullyQualifiedName\": \"OrderImports\",\r\n \"Description\": \"Folder for the inbound order-import automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6110055,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/task.yaml index 70948878f4..8a88ed1947 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-scope-orphan-process-lock/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/README.md new file mode 100644 index 0000000000..7709110944 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/README.md @@ -0,0 +1,61 @@ +# Excel Write Cell — Formula Syntax (Semicolon Separators) + +This scenario reproduces a Write Cell failure where the configured +`Value` is a formula using semicolon parameter separators +(`=SUM(A1;A10)`) instead of the comma separators UiPath / Excel COM +require regardless of host regional setting. The job ends with: + +``` +UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy. +``` + +## What this scenario uncovers + +**Root Cause:** UiPath passes the formula string to Excel COM +verbatim, and Excel COM requires comma separators in parameter +lists regardless of the host's Windows regional setting. The Excel +UI displays semicolons in non-US locales (so workflow authors who +copy formulas from Excel see semicolons and assume they're valid), +but the COM API does not accept them. The formula `=SUM(A1;A10)` +fails to parse, and Excel surfaces the canonical "wrong format, or +Excel is busy" BusinessException. + +**Critical disambiguation:** branch 3 (loop-induced Excel COM +thrash) produces the SAME error wording. The distinguishing +evidence is workflow context: +- This scenario: a single Write Cell with a formula Value, NOT + inside a loop. The user explicitly states "no loop, no parallel + work, no other Excel jobs." +- Branch 3: a Write Cell inside a tight loop where N iterations + succeed before the failure. + +Without checking the workflow source for the formula content AND +the loop context, an agent could pick the wrong branch and +recommend the wrong fix. + +This maps to **branch 2 (formula syntax rejected)** in: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-cell-failures.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelWriteCellProcess` project — `Use Excel File` scope with a single nested `Write Cell` activity whose `Value` is `"=SUM(A1;A10)"` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException wording +identical to branch 3) → `jobs logs` (single Write Cell invocation, +not a sequence of successful iterations) → **workflow source review** +(formula with semicolons + NOT inside a loop = branch 2) → conclude +branch 2, recommend replacing semicolons with commas. + +> **Note on fixtures.** Synthetic. The formula and job key are +> placeholders. The test grades whether the agent reads the +> workflow source, identifies the semicolon separator, and +> contrasts with branch 3 by noting absence of loop — rather than +> reflexively recommending the loop-thrash fix (use Write Range +> instead of Write Cell) which doesn't apply here. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/4417e76e1b.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/4417e76e1b.json deleted file mode 100644 index c0deaa34e1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/4417e76e1b.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "22bb2222-2222-3333-4444-555566667777", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "22bb2222-2222-3333-4444-555566667777", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "22bb2222-2222-3333-4444-555566667777", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/51a2428ded.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/51a2428ded.json deleted file mode 100644 index 7c8f6c3f05..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/51a2428ded.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "22bb2222-2222-3333-4444-555566667777", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9bb22222-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\r\n at UiPath.Excel.WorkbookApplication.SetCellFormula(String sheet, String cell, String formula)\r\n at UiPath.Excel.Activities.WriteCell.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelWriteCellProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5690102, - "OrganizationUnitId": 6020001, - "FolderKey": "f0022222-2222-3333-4444-555566667777", - "Reference": "DailySalesTotal", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "WriteCell_1", - "ActivityDisplayName": "Write Cell: =SUM(A1;A10)", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "22bb2222-2222-3333-4444-555566667777", - "Id": 690099002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/85c5652e29.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/85c5652e29.json deleted file mode 100644 index 2302d185fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/85c5652e29.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-22bb2222-001", - "JobKey": "22bb2222-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-22bb2222-002", - "JobKey": "22bb2222-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelWriteCellProcess] Starting daily total — workbook C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-22bb2222-003", - "JobKey": "22bb2222-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — workbook opened (Modern scope, OpenXML provider)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-22bb2222-004", - "JobKey": "22bb2222-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.110Z", - "Message": "Write Cell: =SUM(A1;A10) — attempting to write formula to Sales!D1", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-22bb2222-005", - "JobKey": "22bb2222-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.310Z", - "Message": "Write Cell: =SUM(A1;A10) — Excel formula parser rejected input", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-22bb2222-006", - "JobKey": "22bb2222-2222-3333-4444-555566667777", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Write Cell: =SUM(A1;A10) — UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/README.md deleted file mode 100644 index 9503b711ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# Excel Write Cell — Formula Syntax (Semicolon Separators) - -This scenario reproduces a Write Cell failure where the configured -`Value` is a formula using semicolon parameter separators -(`=SUM(A1;A10)`) instead of the comma separators UiPath / Excel COM -require regardless of host regional setting. The job ends with: - -``` -UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy. -``` - -## What this scenario uncovers - -**Root Cause:** UiPath passes the formula string to Excel COM -verbatim, and Excel COM requires comma separators in parameter -lists regardless of the host's Windows regional setting. The Excel -UI displays semicolons in non-US locales (so workflow authors who -copy formulas from Excel see semicolons and assume they're valid), -but the COM API does not accept them. The formula `=SUM(A1;A10)` -fails to parse, and Excel surfaces the canonical "wrong format, or -Excel is busy" BusinessException. - -**Critical disambiguation:** branch 3 (loop-induced Excel COM -thrash) produces the SAME error wording. The distinguishing -evidence is workflow context: -- This scenario: a single Write Cell with a formula Value, NOT - inside a loop. The user explicitly states "no loop, no parallel - work, no other Excel jobs." -- Branch 3: a Write Cell inside a tight loop where N iterations - succeed before the failure. - -Without checking the workflow source for the formula content AND -the loop context, an agent could pick the wrong branch and -recommend the wrong fix. - -This maps to **branch 2 (formula syntax rejected)** in: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-cell-failures.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelWriteCellProcess` project — `Use Excel File` scope with a single nested `Write Cell` activity whose `Value` is `"=SUM(A1;A10)"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field is the BusinessException; `or jobs logs` shows the scope opened successfully, then the Write Cell faulted on the first (and only) attempt | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException wording -identical to branch 3) → `jobs logs` (single Write Cell invocation, -not a sequence of successful iterations) → **workflow source review** -(formula with semicolons + NOT inside a loop = branch 2) → conclude -branch 2, recommend replacing semicolons with commas. - -> **Note on fixtures.** Synthetic. The formula and job key are -> placeholders. The test grades whether the agent reads the -> workflow source, identifies the semicolon separator, and -> contrasts with branch 3 by noting absence of loop — rather than -> reflexively recommending the loop-thrash fix (use Write Range -> instead of Write Cell) which doesn't apply here. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/ddc3895174.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/ddc3895174.json deleted file mode 100644 index c40ac8a2e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/ddc3895174.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "22bb2222-2222-3333-4444-555566667777", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelWriteCellProcess", - "Source": "Schedule", - "Reference": "DailySalesTotal", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022222-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/de5ccf6656.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/de5ccf6656.json deleted file mode 100644 index 5192445a32..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/de5ccf6656.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022222-2222-3333-4444-555566667777", - "DisplayName": "ExcelWrites", - "FullyQualifiedName": "ExcelWrites", - "Description": "Folder for daily Excel writes.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/manifest.json deleted file mode 100644 index ecd88ff919..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Write Cell formula scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "de5ccf6656.json"}, - {"match": "or folders list --output json", "file": "de5ccf6656.json"}, - {"match": "or jobs list --folder-key \"f0022222-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "ddc3895174.json"}, - {"match": "or jobs list --folder-key f0022222-2222-3333-4444-555566667777 --state Faulted --output json", "file": "ddc3895174.json"}, - {"match": "or jobs get \"22bb2222-2222-3333-4444-555566667777\" --output json", "file": "51a2428ded.json"}, - {"match": "or jobs get 22bb2222-2222-3333-4444-555566667777 --output json", "file": "51a2428ded.json"}, - {"match": "or jobs history \"22bb2222-2222-3333-4444-555566667777\" --output json", "file": "4417e76e1b.json"}, - {"match": "or jobs history 22bb2222-2222-3333-4444-555566667777 --output json", "file": "4417e76e1b.json"}, - {"match": "or jobs logs \"22bb2222-2222-3333-4444-555566667777\" --output json", "file": "85c5652e29.json"}, - {"match": "or jobs logs 22bb2222-2222-3333-4444-555566667777 --output json", "file": "85c5652e29.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/uip-fixture.json new file mode 100644 index 0000000000..b2d0d15238 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0022222-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelWriteCellProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailySalesTotal\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "22bb2222-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9bb22222-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\\r\\n at UiPath.Excel.WorkbookApplication.SetCellFormula(String sheet, String cell, String formula)\\r\\n at UiPath.Excel.Activities.WriteCell.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelWriteCellProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5690102,\r\n \"OrganizationUnitId\": 6020001,\r\n \"FolderKey\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"Reference\": \"DailySalesTotal\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"WriteCell_1\",\r\n \"ActivityDisplayName\": \"Write Cell: =SUM(A1;A10)\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Id\": 690099002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "22bb2222-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"22bb2222-2222-3333-4444-555566667777\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"22bb2222-2222-3333-4444-555566667777\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"22bb2222-2222-3333-4444-555566667777\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "22bb2222-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-22bb2222-001\",\r\n \"JobKey\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-22bb2222-002\",\r\n \"JobKey\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelWriteCellProcess] Starting daily total — workbook C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-22bb2222-003\",\r\n \"JobKey\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — workbook opened (Modern scope, OpenXML provider)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-22bb2222-004\",\r\n \"JobKey\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.110Z\",\r\n \"Message\": \"Write Cell: =SUM(A1;A10) — attempting to write formula to Sales!D1\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-22bb2222-005\",\r\n \"JobKey\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.310Z\",\r\n \"Message\": \"Write Cell: =SUM(A1;A10) — Excel formula parser rejected input\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-22bb2222-006\",\r\n \"JobKey\": \"22bb2222-2222-3333-4444-555566667777\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Write Cell: =SUM(A1;A10) — UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelWrites\",\r\n \"FullyQualifiedName\": \"ExcelWrites\",\r\n \"Description\": \"Folder for daily Excel writes.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/task.yaml index 8da8ff5e5a..38267f7253 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-formula-semicolons/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/README.md new file mode 100644 index 0000000000..bfb630dc10 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/README.md @@ -0,0 +1,70 @@ +# Excel Write Cell — Loop-Induced "Excel Is Busy" + +This scenario reproduces a Write Cell failure that emerges only +after N successful iterations inside a `For Each Row` loop. The +job ends with the canonical: + +``` +UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy. +``` + +— same wording as branch 2 (formula syntax) but with the "Excel +is busy" clause actually applicable here. + +## What this scenario uncovers + +**Root Cause:** The workflow's `Write Cell` activity is nested +inside a `For Each Row` loop. Each iteration opens the workbook, +writes one cell, saves, and closes the file. After N iterations +(247 in this job, different on retry) Excel COM state becomes +corrupt — COM-object leaks accumulate, the file-lock churn races +with itself, and the activity faults. + +The "or Excel is busy" clause of the error is misleading on +branch 2 (formula syntax — nothing is busy) but ACCURATE on +branch 3: Excel COM is in a degraded state from the open/save/ +close churn. + +**Critical disambiguation from branch 2:** the same error wording +is produced by formula syntax errors. The distinguishing evidence +is workflow context + execution pattern: + +| Signal | Branch 2 (formula syntax) | Branch 3 (loop thrash) | +|---|---|---| +| Workflow source | Single Write Cell, no loop | Write Cell inside For Each (Row) | +| Failure pattern | Fails on first iteration, deterministic | N iterations succeed, then fails; N varies between runs | +| User context | "Single activity, no loop" | "Hundreds of rows wrote before failure" | +| Job logs | One Write Cell attempt, fails | Many successful Write Cell invocations preceding the failure | + +This maps to **branch 3 (loop-induced Excel-is-busy / COM thrash)** +in: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-cell-failures.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelWriteCellProcess` project — `Use Excel File` scope opens the workbook, `Read Range` populates a DataTable, then `For Each Row` iterates the table writing one cell per row via `Write Cell` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException +identical to branch 2) → `jobs logs` (**N successful Write Cell +iterations**, then fault on iteration N+1 — the partial-success +pattern is the smoking gun) → **workflow source review** (For Each +Row + Write Cell inside) → conclude branch 3, recommend bulk +Read Range → DataTable → Write Range pattern. + +> **Note on fixtures.** Synthetic. The exact iteration count (247), +> column names, and workbook structure are placeholders. The test +> grades whether the agent: +> +> 1. Notices the partial-success pattern in job logs (rules out +> branch 2). +> 2. Identifies the For Each Row + Write Cell-inside pattern in the +> workflow source (positive signal for branch 3). +> 3. Recommends the bulk pattern fix — NOT Retry Scope, Delay, +> or formula-syntax changes (all documented anti-patterns). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/606ce5b486.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/606ce5b486.json deleted file mode 100644 index 258cffe78b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/606ce5b486.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "33cc3333-2222-3333-4444-555566667777", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:04:18.412Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9cc33333-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\r\n at UiPath.Excel.WorkbookApplication.SetCellValue(String sheet, String cell, Object value)\r\n at UiPath.Excel.Activities.WriteCell.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelWriteCellProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5690103, - "OrganizationUnitId": 6020001, - "FolderKey": "f0022222-2222-3333-4444-555566667777", - "Reference": "DailyStatusUpdate", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:04:18.412Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "WriteCell_1", - "ActivityDisplayName": "Write Cell: row N status", - "WorkflowFilePath": "Main.xaml", - "FailedIteration": 248 - }, - "ErrorCode": "Activity", - "TraceId": "33cc3333-2222-3333-4444-555566667777", - "Id": 690099003 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/README.md deleted file mode 100644 index 464fdb12c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Excel Write Cell — Loop-Induced "Excel Is Busy" - -This scenario reproduces a Write Cell failure that emerges only -after N successful iterations inside a `For Each Row` loop. The -job ends with the canonical: - -``` -UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy. -``` - -— same wording as branch 2 (formula syntax) but with the "Excel -is busy" clause actually applicable here. - -## What this scenario uncovers - -**Root Cause:** The workflow's `Write Cell` activity is nested -inside a `For Each Row` loop. Each iteration opens the workbook, -writes one cell, saves, and closes the file. After N iterations -(247 in this job, different on retry) Excel COM state becomes -corrupt — COM-object leaks accumulate, the file-lock churn races -with itself, and the activity faults. - -The "or Excel is busy" clause of the error is misleading on -branch 2 (formula syntax — nothing is busy) but ACCURATE on -branch 3: Excel COM is in a degraded state from the open/save/ -close churn. - -**Critical disambiguation from branch 2:** the same error wording -is produced by formula syntax errors. The distinguishing evidence -is workflow context + execution pattern: - -| Signal | Branch 2 (formula syntax) | Branch 3 (loop thrash) | -|---|---|---| -| Workflow source | Single Write Cell, no loop | Write Cell inside For Each (Row) | -| Failure pattern | Fails on first iteration, deterministic | N iterations succeed, then fails; N varies between runs | -| User context | "Single activity, no loop" | "Hundreds of rows wrote before failure" | -| Job logs | One Write Cell attempt, fails | Many successful Write Cell invocations preceding the failure | - -This maps to **branch 3 (loop-induced Excel-is-busy / COM thrash)** -in: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-cell-failures.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelWriteCellProcess` project — `Use Excel File` scope opens the workbook, `Read Range` populates a DataTable, then `For Each Row` iterates the table writing one cell per row via `Write Cell` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field is the BusinessException; `or jobs logs` shows ~247 successful Write Cell iterations preceding the failure | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException -identical to branch 2) → `jobs logs` (**N successful Write Cell -iterations**, then fault on iteration N+1 — the partial-success -pattern is the smoking gun) → **workflow source review** (For Each -Row + Write Cell inside) → conclude branch 3, recommend bulk -Read Range → DataTable → Write Range pattern. - -> **Note on fixtures.** Synthetic. The exact iteration count (247), -> column names, and workbook structure are placeholders. The test -> grades whether the agent: -> -> 1. Notices the partial-success pattern in job logs (rules out -> branch 2). -> 2. Identifies the For Each Row + Write Cell-inside pattern in the -> workflow source (positive signal for branch 3). -> 3. Recommends the bulk pattern fix — NOT Retry Scope, Delay, -> or formula-syntax changes (all documented anti-patterns). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/a99f8286f1.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/a99f8286f1.json deleted file mode 100644 index 6c814f2223..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/a99f8286f1.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "33cc3333-2222-3333-4444-555566667777", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:04:18.412Z", - "State": "Faulted", - "ReleaseName": "ExcelWriteCellProcess", - "Source": "Schedule", - "Reference": "DailyStatusUpdate", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022222-2222-3333-4444-555566667777", - "Info": "UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/b67fb93b82.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/b67fb93b82.json deleted file mode 100644 index cef688f17d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/b67fb93b82.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "33cc3333-2222-3333-4444-555566667777", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "33cc3333-2222-3333-4444-555566667777", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "33cc3333-2222-3333-4444-555566667777", "State": "Faulted", "Timestamp": "2026-05-20T08:04:18.412Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/de5ccf6656.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/de5ccf6656.json deleted file mode 100644 index 5192445a32..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/de5ccf6656.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022222-2222-3333-4444-555566667777", - "DisplayName": "ExcelWrites", - "FullyQualifiedName": "ExcelWrites", - "Description": "Folder for daily Excel writes.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/f481fd3c02.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/f481fd3c02.json deleted file mode 100644 index eea6db9f48..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/f481fd3c02.json +++ /dev/null @@ -1,166 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-33cc3333-001", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-002", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelWriteCellProcess] Per-row status update — workbook C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-003", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — workbook opened (Modern scope, OpenXML provider)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-004", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.812Z", - "Message": "Read Range: A1:F313 — completed, 312 rows read into Rows", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-005", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.110Z", - "Message": "For Each Row in Rows — iteration 1 of 312", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-006", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.312Z", - "Message": "Write Cell: row N status — iteration 1: wrote 'Normal' to Sales!E2", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-007", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.512Z", - "Message": "Write Cell: row N status — iteration 2: wrote 'High' to Sales!E3", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-008", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:12.310Z", - "Message": "Write Cell: row N status — iteration 50: wrote 'Normal' to Sales!E51 (~200ms/iter avg)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-009", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:22.418Z", - "Message": "Write Cell: row N status — iteration 100: wrote 'High' to Sales!E101 (~210ms/iter avg)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-010", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:01:18.512Z", - "Message": "Write Cell: row N status — iteration 150: wrote 'Normal' to Sales!E151 (~270ms/iter avg, perf degrading)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-011", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:02:31.812Z", - "Message": "Write Cell: row N status — iteration 200: wrote 'High' to Sales!E201 (~440ms/iter avg, perf degrading)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-012", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:04:02.312Z", - "Message": "Write Cell: row N status — iteration 245: wrote 'Normal' to Sales!E246 (~860ms/iter avg, perf degrading)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-013", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:04:06.118Z", - "Message": "Write Cell: row N status — iteration 246: wrote 'High' to Sales!E247", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-014", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:04:09.910Z", - "Message": "Write Cell: row N status — iteration 247: wrote 'Normal' to Sales!E248", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-015", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:04:13.610Z", - "Message": "For Each Row in Rows — iteration 248 of 312", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-33cc3333-016", - "JobKey": "33cc3333-2222-3333-4444-555566667777", - "Level": "Error", - "Timestamp": "2026-05-20T08:04:18.402Z", - "Message": "Write Cell: row N status — iteration 248: UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/manifest.json deleted file mode 100644 index 2e64c097e3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Write Cell per-row status scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "de5ccf6656.json"}, - {"match": "or folders list --output json", "file": "de5ccf6656.json"}, - {"match": "or jobs list --folder-key \"f0022222-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "a99f8286f1.json"}, - {"match": "or jobs list --folder-key f0022222-2222-3333-4444-555566667777 --state Faulted --output json", "file": "a99f8286f1.json"}, - {"match": "or jobs get \"33cc3333-2222-3333-4444-555566667777\" --output json", "file": "606ce5b486.json"}, - {"match": "or jobs get 33cc3333-2222-3333-4444-555566667777 --output json", "file": "606ce5b486.json"}, - {"match": "or jobs history \"33cc3333-2222-3333-4444-555566667777\" --output json", "file": "b67fb93b82.json"}, - {"match": "or jobs history 33cc3333-2222-3333-4444-555566667777 --output json", "file": "b67fb93b82.json"}, - {"match": "or jobs logs \"33cc3333-2222-3333-4444-555566667777\" --output json", "file": "f481fd3c02.json"}, - {"match": "or jobs logs 33cc3333-2222-3333-4444-555566667777 --output json", "file": "f481fd3c02.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/uip-fixture.json new file mode 100644 index 0000000000..34ad0514ae --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0022222-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:04:18.412Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelWriteCellProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailyStatusUpdate\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "33cc3333-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:04:18.412Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9cc33333-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\\r\\n at UiPath.Excel.WorkbookApplication.SetCellValue(String sheet, String cell, Object value)\\r\\n at UiPath.Excel.Activities.WriteCell.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelWriteCellProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5690103,\r\n \"OrganizationUnitId\": 6020001,\r\n \"FolderKey\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"Reference\": \"DailyStatusUpdate\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:04:18.412Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"WriteCell_1\",\r\n \"ActivityDisplayName\": \"Write Cell: row N status\",\r\n \"WorkflowFilePath\": \"Main.xaml\",\r\n \"FailedIteration\": 248\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Id\": 690099003\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "33cc3333-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"33cc3333-2222-3333-4444-555566667777\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"33cc3333-2222-3333-4444-555566667777\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"33cc3333-2222-3333-4444-555566667777\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:04:18.412Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "33cc3333-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-33cc3333-001\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-002\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelWriteCellProcess] Per-row status update — workbook C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-003\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — workbook opened (Modern scope, OpenXML provider)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-004\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.812Z\",\r\n \"Message\": \"Read Range: A1:F313 — completed, 312 rows read into Rows\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-005\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.110Z\",\r\n \"Message\": \"For Each Row in Rows — iteration 1 of 312\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-006\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.312Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 1: wrote 'Normal' to Sales!E2\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-007\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.512Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 2: wrote 'High' to Sales!E3\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-008\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:12.310Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 50: wrote 'Normal' to Sales!E51 (~200ms/iter avg)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-009\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:22.418Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 100: wrote 'High' to Sales!E101 (~210ms/iter avg)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-010\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:01:18.512Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 150: wrote 'Normal' to Sales!E151 (~270ms/iter avg, perf degrading)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-011\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:02:31.812Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 200: wrote 'High' to Sales!E201 (~440ms/iter avg, perf degrading)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-012\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:04:02.312Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 245: wrote 'Normal' to Sales!E246 (~860ms/iter avg, perf degrading)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-013\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:04:06.118Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 246: wrote 'High' to Sales!E247\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-014\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:04:09.910Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 247: wrote 'Normal' to Sales!E248\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-015\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:04:13.610Z\",\r\n \"Message\": \"For Each Row in Rows — iteration 248 of 312\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-33cc3333-016\",\r\n \"JobKey\": \"33cc3333-2222-3333-4444-555566667777\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:04:18.402Z\",\r\n \"Message\": \"Write Cell: row N status — iteration 248: UiPath.Excel.BusinessException: The data you want to write has a wrong format, or Excel is busy.\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelWrites\",\r\n \"FullyQualifiedName\": \"ExcelWrites\",\r\n \"Description\": \"Folder for daily Excel writes.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/task.yaml index 3e2dbc3f4a..e90ed8b798 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-loop-thrash/task.yaml @@ -30,21 +30,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/README.md new file mode 100644 index 0000000000..619f10bdcb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/README.md @@ -0,0 +1,61 @@ +# Excel Write Cell — Classic/Modern Scope Conflict + +This scenario reproduces a Write Cell failure with the canonical +"file in use by another process" `IOException` — the same wording +as the Read Range file-locked playbook — but here the locker is +NOT external. The workflow's own surrounding Modern `Use Excel File` +scope holds the file while a nested Classic Workbook `Write Cell` +activity tries to write to the same file. The job ends with: + +``` +System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow contains a Modern `Use Excel File` +scope on `C:\Robot\Data\sales-2026-05.xlsx`, and INSIDE its body +a Classic Workbook `Write Cell` activity with its own +`WorkbookPath` property pointing at the same file. The Classic +Workbook surface accesses the file's raw bytes directly and +refuses to write while another process — including UiPath's own +Modern scope — has the file open. + +The error wording is identical to an external-locker failure +(orphan `EXCEL.EXE`, user editing, network share, etc.) but the +user has explicitly ruled those out. The workflow source is what +makes the diagnosis: two scopes / surfaces on the same file +inside the same job. + +This maps to the **scope-conflict variant of branch 1** in: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-cell-failures.md` + +The agent must distinguish this from the **external-locker variant** +covered by `workbook-file-locked.md`. The decisive signal is the +workflow source — both scopes visible in `Main.xaml` — combined +with the user's explicit exclusion of external lockers. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelWriteCellProcess` project — `Use Excel File` scope opens `sales-2026-05.xlsx`; inside the scope's body, a Classic Workbook `Write Cell` targets the same file | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (IOException) → +`jobs logs` (Modern scope opened the file successfully; Classic +Workbook activity faulted next) → **workflow source review** (the +dual-scope pattern is the smoking gun) → conclude branch 1 +scope-conflict variant. + +> **Note on fixtures.** Synthetic. The workbook path and job key +> are placeholders. The test grades whether the agent reads the +> workflow source, spots the dual-scope pattern, and recommends +> either nesting Modern Write Cell or unnesting the Classic +> activity — rather than reflexively pivoting to the read-range +> file-locked investigation chain (orphan check, network share, +> AV scanner) which the user has explicitly ruled out. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/5b1b798a5a.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/5b1b798a5a.json deleted file mode 100644 index c809981647..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/5b1b798a5a.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-11aa1111-001", - "JobKey": "11aa1111-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-11aa1111-002", - "JobKey": "11aa1111-2222-3333-4444-555566667777", - "Level": "Info", - "Timestamp": "2026-05-20T08:00:01.418Z", - "Message": "[ExcelWriteCellProcess] Starting daily write — workbook C:\\Robot\\Data\\sales-2026-05.xlsx", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-11aa1111-003", - "JobKey": "11aa1111-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.510Z", - "Message": "Use Excel File: sales-2026-05.xlsx — workbook opened (Modern scope, OpenXML provider)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-11aa1111-004", - "JobKey": "11aa1111-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:01.812Z", - "Message": "Read Range: A1:B10 (Modern) — completed, 10 rows read into ReadData", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-11aa1111-005", - "JobKey": "11aa1111-2222-3333-4444-555566667777", - "Level": "Trace", - "Timestamp": "2026-05-20T08:00:02.110Z", - "Message": "Write Cell: status=Done (Classic Workbook surface) — opening workbook C:\\Robot\\Data\\sales-2026-05.xlsx for direct write (Classic surface does not delegate to surrounding scope)", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-11aa1111-006", - "JobKey": "11aa1111-2222-3333-4444-555566667777", - "Level": "Error", - "Timestamp": "2026-05-20T08:00:02.802Z", - "Message": "Write Cell: status=Done (Classic Workbook surface) — System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ProcessName": "ExcelWriteCellProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/6e024aedde.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/6e024aedde.json deleted file mode 100644 index f24932a9cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/6e024aedde.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "11aa1111-2222-3333-4444-555566667777", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelWriteCellProcess", - "Source": "Schedule", - "Reference": "DailySalesWrite", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022222-2222-3333-4444-555566667777", - "Info": "System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/README.md deleted file mode 100644 index a7f578c7e1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# Excel Write Cell — Classic/Modern Scope Conflict - -This scenario reproduces a Write Cell failure with the canonical -"file in use by another process" `IOException` — the same wording -as the Read Range file-locked playbook — but here the locker is -NOT external. The workflow's own surrounding Modern `Use Excel File` -scope holds the file while a nested Classic Workbook `Write Cell` -activity tries to write to the same file. The job ends with: - -``` -System.IO.IOException: The process cannot access the file 'C:\Robot\Data\sales-2026-05.xlsx' because it is being used by another process. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow contains a Modern `Use Excel File` -scope on `C:\Robot\Data\sales-2026-05.xlsx`, and INSIDE its body -a Classic Workbook `Write Cell` activity with its own -`WorkbookPath` property pointing at the same file. The Classic -Workbook surface accesses the file's raw bytes directly and -refuses to write while another process — including UiPath's own -Modern scope — has the file open. - -The error wording is identical to an external-locker failure -(orphan `EXCEL.EXE`, user editing, network share, etc.) but the -user has explicitly ruled those out. The workflow source is what -makes the diagnosis: two scopes / surfaces on the same file -inside the same job. - -This maps to the **scope-conflict variant of branch 1** in: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-cell-failures.md` - -The agent must distinguish this from the **external-locker variant** -covered by `workbook-file-locked.md`. The decisive signal is the -workflow source — both scopes visible in `Main.xaml` — combined -with the user's explicit exclusion of external lockers. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelWriteCellProcess` project — `Use Excel File` scope opens `sales-2026-05.xlsx`; inside the scope's body, a Classic Workbook `Write Cell` targets the same file | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs get` Info field is the IOException; `or jobs logs` shows the scope opening succeeded, then the Classic Workbook Write Cell faulted | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (IOException) → -`jobs logs` (Modern scope opened the file successfully; Classic -Workbook activity faulted next) → **workflow source review** (the -dual-scope pattern is the smoking gun) → conclude branch 1 -scope-conflict variant. - -> **Note on fixtures.** Synthetic. The workbook path and job key -> are placeholders. The test grades whether the agent reads the -> workflow source, spots the dual-scope pattern, and recommends -> either nesting Modern Write Cell or unnesting the Classic -> activity — rather than reflexively pivoting to the read-range -> file-locked investigation chain (orphan check, network share, -> AV scanner) which the user has explicitly ruled out. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/cf0807f3dd.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/cf0807f3dd.json deleted file mode 100644 index 74f9b392c8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/cf0807f3dd.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "11aa1111-2222-3333-4444-555566667777", "State": "Pending", "Timestamp": "2026-05-20T08:00:01.005Z"}, - {"Key": "11aa1111-2222-3333-4444-555566667777", "State": "Running", "Timestamp": "2026-05-20T08:00:01.300Z"}, - {"Key": "11aa1111-2222-3333-4444-555566667777", "State": "Faulted", "Timestamp": "2026-05-20T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/de5ccf6656.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/de5ccf6656.json deleted file mode 100644 index 5192445a32..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/de5ccf6656.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022222-2222-3333-4444-555566667777", - "DisplayName": "ExcelWrites", - "FullyQualifiedName": "ExcelWrites", - "Description": "Folder for daily Excel writes.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/ecd9fb418d.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/ecd9fb418d.json deleted file mode 100644 index aa75de20a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/ecd9fb418d.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "11aa1111-2222-3333-4444-555566667777", - "StartTime": "2026-05-20T08:00:01.300Z", - "EndTime": "2026-05-20T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9aa11111-2222-3333-4444-555566667777", - "Info": "System.IO.IOException: The process cannot access the file 'C:\\Robot\\Data\\sales-2026-05.xlsx' because it is being used by another process.\r\n at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)\r\n at UiPath.Excel.Activities.WriteCell.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-20T08:00:01.005Z", - "ReleaseName": "ExcelWriteCellProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5690101, - "OrganizationUnitId": 6020001, - "FolderKey": "f0022222-2222-3333-4444-555566667777", - "Reference": "DailySalesWrite", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-20T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "IOException", - "ActivityName": "WriteCell_1", - "ActivityDisplayName": "Write Cell: status=Done (Classic Workbook surface)", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "11aa1111-2222-3333-4444-555566667777", - "Id": 690099001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/manifest.json deleted file mode 100644 index a5bb4e32e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Excel Write Cell status-update scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "de5ccf6656.json"}, - {"match": "or folders list --output json", "file": "de5ccf6656.json"}, - {"match": "or jobs list --folder-key \"f0022222-2222-3333-4444-555566667777\" --state Faulted --output json", "file": "6e024aedde.json"}, - {"match": "or jobs list --folder-key f0022222-2222-3333-4444-555566667777 --state Faulted --output json", "file": "6e024aedde.json"}, - {"match": "or jobs get \"11aa1111-2222-3333-4444-555566667777\" --output json", "file": "ecd9fb418d.json"}, - {"match": "or jobs get 11aa1111-2222-3333-4444-555566667777 --output json", "file": "ecd9fb418d.json"}, - {"match": "or jobs history \"11aa1111-2222-3333-4444-555566667777\" --output json", "file": "cf0807f3dd.json"}, - {"match": "or jobs history 11aa1111-2222-3333-4444-555566667777 --output json", "file": "cf0807f3dd.json"}, - {"match": "or jobs logs \"11aa1111-2222-3333-4444-555566667777\" --output json", "file": "5b1b798a5a.json"}, - {"match": "or jobs logs 11aa1111-2222-3333-4444-555566667777 --output json", "file": "5b1b798a5a.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/uip-fixture.json new file mode 100644 index 0000000000..3e85ed2a0b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0022222-2222-3333-4444-555566667777", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelWriteCellProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailySalesWrite\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "11aa1111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"StartTime\": \"2026-05-20T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9aa11111-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\\r\\n at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)\\r\\n at UiPath.Excel.Activities.WriteCell.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-20T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelWriteCellProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5690101,\r\n \"OrganizationUnitId\": 6020001,\r\n \"FolderKey\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"Reference\": \"DailySalesWrite\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-20T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"IOException\",\r\n \"ActivityName\": \"WriteCell_1\",\r\n \"ActivityDisplayName\": \"Write Cell: status=Done (Classic Workbook surface)\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Id\": 690099001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "11aa1111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"11aa1111-2222-3333-4444-555566667777\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-20T08:00:01.005Z\"},\r\n {\"Key\": \"11aa1111-2222-3333-4444-555566667777\", \"State\": \"Running\", \"Timestamp\": \"2026-05-20T08:00:01.300Z\"},\r\n {\"Key\": \"11aa1111-2222-3333-4444-555566667777\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-20T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "11aa1111-2222-3333-4444-555566667777", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-11aa1111-001\",\r\n \"JobKey\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-11aa1111-002\",\r\n \"JobKey\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.418Z\",\r\n \"Message\": \"[ExcelWriteCellProcess] Starting daily write — workbook C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-11aa1111-003\",\r\n \"JobKey\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: sales-2026-05.xlsx — workbook opened (Modern scope, OpenXML provider)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-11aa1111-004\",\r\n \"JobKey\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:01.812Z\",\r\n \"Message\": \"Read Range: A1:B10 (Modern) — completed, 10 rows read into ReadData\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-11aa1111-005\",\r\n \"JobKey\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.110Z\",\r\n \"Message\": \"Write Cell: status=Done (Classic Workbook surface) — opening workbook C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx for direct write (Classic surface does not delegate to surrounding scope)\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-11aa1111-006\",\r\n \"JobKey\": \"11aa1111-2222-3333-4444-555566667777\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-20T08:00:02.802Z\",\r\n \"Message\": \"Write Cell: status=Done (Classic Workbook surface) — System.IO.IOException: The process cannot access the file 'C:\\\\Robot\\\\Data\\\\sales-2026-05.xlsx' because it is being used by another process.\",\r\n \"ProcessName\": \"ExcelWriteCellProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022222-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"ExcelWrites\",\r\n \"FullyQualifiedName\": \"ExcelWrites\",\r\n \"Description\": \"Folder for daily Excel writes.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/task.yaml index 2a1ce0e94b..5a5aa262c4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-wc-scope-conflict/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/README.md new file mode 100644 index 0000000000..b7f50e061b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/README.md @@ -0,0 +1,57 @@ +# Excel Write Range — Empty Source DataTable + +This scenario reproduces a Write Range failure where the source +`DataTable` has 0 rows (a Filter DataTable removed everything) and +`ExcludeHeaders=False`. The job ends with: + +``` +UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's Write Range received an initialized +DataTable (`dtOverdue`) with 0 rows because the preceding Filter +DataTable activity removed every row — today's batch of 247 invoices +contained no rows with `Status="Overdue"`. With `ExcludeHeaders=False` +(the default), Write Range treats a 0-row source as a configuration +error rather than a no-op and throws. The "Ignore empty source" +property in the error message is misleading — it only handles +`Nothing` DataTables (playbook branch 1), not initialized 0-row +DataTables. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-range-failures.md` +(the "Empty source + ExcludeHeaders=False" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelInvoiceProcess` project — `Read Range` → `Filter DataTable: Status = Overdue` → `Write Range` with `ExcludeHeaders=False` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (BusinessException +referencing "Ignore empty source is ineffective" on WriteRange_1) → +`jobs logs` (the Filter DataTable 247→0 Trace + the WriteRange +`rows=0 cols=8 ExcludeHeaders=False` config Trace are the smoking +gun) → workflow source review (confirms no `If rows.Count > 0` +guard around the Write Range) → conclude branch 3. + +> **Note on the anti-pattern trap.** The error message names +> "Exclude headers" as the relevant property, which can mislead +> the agent into recommending `ExcludeHeaders=True` as the fix. +> That's wrong — it would silence the error but produce wrong-shape +> writes (data row in header position). The correct fix is an +> `If dtOverdue.Rows.Count > 0` guard. The test rewards agents +> that recommend the guard and penalizes the ExcludeHeaders silencer. + +> **Note on fixtures.** Synthetic. The status breakdown (247 / 189 / +> 58 / 0), filter predicate, and timing values are placeholders. +> The test grades whether the agent correctly diagnoses the 0-row +> case + ExcludeHeaders=False combination AND recommends the +> If-guard fix (not the ExcludeHeaders silencer). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/223dcd840f.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/223dcd840f.json deleted file mode 100644 index 1db91298c6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/223dcd840f.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ee2222cc-001", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-25T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-002", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-25T08:00:01.418Z", - "Message": "[ExcelInvoiceProcess] Filtering overdue invoices and writing report to C:\\Robot\\Data\\invoice-report.xlsx", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-003", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-25T08:00:01.510Z", - "Message": "Use Excel File: invoices-source.xlsx — workbook opened (OpenXML provider)", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-004", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-25T08:00:01.810Z", - "Message": "Read Range: load invoices (Invoices!A1:Z10000) — returned 247 rows (Status values: Paid=189, Pending=58, Overdue=0)", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-005", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-25T08:00:02.010Z", - "Message": "Filter Data Table: Status = Overdue — input 247 rows, output 0 rows (no matches)", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-006", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-25T08:00:02.310Z", - "Message": "Use Excel File: invoice-report.xlsx — workbook opened (OpenXML provider)", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-007", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-25T08:00:02.510Z", - "Message": "Write Range: overdue report — resolved config Sheet='Overdue' StartingCell='A1' ExcludeHeaders=False DataTable='dtOverdue' (rows=0 cols=8)", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee2222cc-008", - "JobKey": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-25T08:00:02.802Z", - "Message": "Write Range: overdue report — UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False.", - "ProcessName": "ExcelInvoiceProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/4b0d4c39e8.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/4b0d4c39e8.json deleted file mode 100644 index 5dbaf59ee8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/4b0d4c39e8.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-25T08:00:01.300Z", - "EndTime": "2026-05-25T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelInvoiceProcess", - "Source": "Schedule", - "Reference": "DailyOverdueA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0066666-3333-4444-5555-666677778888", - "Info": "UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/README.md deleted file mode 100644 index e4ece711f0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/README.md +++ /dev/null @@ -1,56 +0,0 @@ -# Excel Write Range — Empty Source DataTable - -This scenario reproduces a Write Range failure where the source -`DataTable` has 0 rows (a Filter DataTable removed everything) and -`ExcludeHeaders=False`. The job ends with: - -``` -UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's Write Range received an initialized -DataTable (`dtOverdue`) with 0 rows because the preceding Filter -DataTable activity removed every row — today's batch of 247 invoices -contained no rows with `Status="Overdue"`. With `ExcludeHeaders=False` -(the default), Write Range treats a 0-row source as a configuration -error rather than a no-op and throws. The "Ignore empty source" -property in the error message is misleading — it only handles -`Nothing` DataTables (playbook branch 1), not initialized 0-row -DataTables. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-range-failures.md` -(the "Empty source + ExcludeHeaders=False" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelInvoiceProcess` project — `Read Range` → `Filter DataTable: Status = Overdue` → `Write Range` with `ExcludeHeaders=False` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture the 247→0 filter drop and the WriteRange resolved-config showing `rows=0 cols=8 ExcludeHeaders=False` | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (BusinessException -referencing "Ignore empty source is ineffective" on WriteRange_1) → -`jobs logs` (the Filter DataTable 247→0 Trace + the WriteRange -`rows=0 cols=8 ExcludeHeaders=False` config Trace are the smoking -gun) → workflow source review (confirms no `If rows.Count > 0` -guard around the Write Range) → conclude branch 3. - -> **Note on the anti-pattern trap.** The error message names -> "Exclude headers" as the relevant property, which can mislead -> the agent into recommending `ExcludeHeaders=True` as the fix. -> That's wrong — it would silence the error but produce wrong-shape -> writes (data row in header position). The correct fix is an -> `If dtOverdue.Rows.Count > 0` guard. The test rewards agents -> that recommend the guard and penalizes the ExcludeHeaders silencer. - -> **Note on fixtures.** Synthetic. The status breakdown (247 / 189 / -> 58 / 0), filter predicate, and timing values are placeholders. -> The test grades whether the agent correctly diagnoses the 0-row -> case + ExcludeHeaders=False combination AND recommends the -> If-guard fix (not the ExcludeHeaders silencer). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/b78282fe56.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/b78282fe56.json deleted file mode 100644 index 8cee404686..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/b78282fe56.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-25T08:00:01.005Z"}, - {"Key": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-25T08:00:01.300Z"}, - {"Key": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-25T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/d665273ce3.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/d665273ce3.json deleted file mode 100644 index 4363eb5bfa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/d665273ce3.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-25T08:00:01.300Z", - "EndTime": "2026-05-25T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ee22222-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False.\r\n at UiPath.Excel.Activities.Business.WriteRangeX.ValidateSourceTable(DataTable source)\r\n at UiPath.Excel.Activities.Business.WriteRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-25T08:00:01.005Z", - "ReleaseName": "ExcelInvoiceProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680601, - "OrganizationUnitId": 6060001, - "FolderKey": "f0066666-3333-4444-5555-666677778888", - "Reference": "DailyOverdueA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-25T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "WriteRange_1", - "ActivityDisplayName": "Write Range: overdue report", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ee222222-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680699001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/d7aedb669e.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/d7aedb669e.json deleted file mode 100644 index 38e3367388..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/d7aedb669e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0066666-3333-4444-5555-666677778888", - "DisplayName": "InvoiceOps", - "FullyQualifiedName": "InvoiceOps", - "Description": "Folder for daily invoice-reporting automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6060001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/manifest.json deleted file mode 100644 index acf315cb5f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Write Range overdue-invoice report scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "d7aedb669e.json"}, - {"match": "or folders list --output json", "file": "d7aedb669e.json"}, - {"match": "or jobs list --folder-key \"f0066666-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "4b0d4c39e8.json"}, - {"match": "or jobs list --folder-key f0066666-3333-4444-5555-666677778888 --state Faulted --output json", "file": "4b0d4c39e8.json"}, - {"match": "or jobs get \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "d665273ce3.json"}, - {"match": "or jobs get ee222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "d665273ce3.json"}, - {"match": "or jobs history \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "b78282fe56.json"}, - {"match": "or jobs history ee222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "b78282fe56.json"}, - {"match": "or jobs logs \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "223dcd840f.json"}, - {"match": "or jobs logs ee222222-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "223dcd840f.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/uip-fixture.json new file mode 100644 index 0000000000..a1aaa1fb51 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0066666-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-25T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-25T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelInvoiceProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"DailyOverdueA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0066666-3333-4444-5555-666677778888\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ee222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-25T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-25T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ee22222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False.\\r\\n at UiPath.Excel.Activities.Business.WriteRangeX.ValidateSourceTable(DataTable source)\\r\\n at UiPath.Excel.Activities.Business.WriteRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-25T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelInvoiceProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680601,\r\n \"OrganizationUnitId\": 6060001,\r\n \"FolderKey\": \"f0066666-3333-4444-5555-666677778888\",\r\n \"Reference\": \"DailyOverdueA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-25T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"WriteRange_1\",\r\n \"ActivityDisplayName\": \"Write Range: overdue report\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680699001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ee222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-25T08:00:01.005Z\"},\r\n {\"Key\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-25T08:00:01.300Z\"},\r\n {\"Key\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-25T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ee222222-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ee2222cc-001\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-25T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-002\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-25T08:00:01.418Z\",\r\n \"Message\": \"[ExcelInvoiceProcess] Filtering overdue invoices and writing report to C:\\\\Robot\\\\Data\\\\invoice-report.xlsx\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-003\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-25T08:00:01.510Z\",\r\n \"Message\": \"Use Excel File: invoices-source.xlsx — workbook opened (OpenXML provider)\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-004\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-25T08:00:01.810Z\",\r\n \"Message\": \"Read Range: load invoices (Invoices!A1:Z10000) — returned 247 rows (Status values: Paid=189, Pending=58, Overdue=0)\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-005\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-25T08:00:02.010Z\",\r\n \"Message\": \"Filter Data Table: Status = Overdue — input 247 rows, output 0 rows (no matches)\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-006\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-25T08:00:02.310Z\",\r\n \"Message\": \"Use Excel File: invoice-report.xlsx — workbook opened (OpenXML provider)\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-007\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-25T08:00:02.510Z\",\r\n \"Message\": \"Write Range: overdue report — resolved config Sheet='Overdue' StartingCell='A1' ExcludeHeaders=False DataTable='dtOverdue' (rows=0 cols=8)\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee2222cc-008\",\r\n \"JobKey\": \"ee222222-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-25T08:00:02.802Z\",\r\n \"Message\": \"Write Range: overdue report — UiPath.Excel.BusinessException: The Excel Activity option 'Ignore empty source' is ineffective: the source DataTable has 0 rows and 'Exclude headers' is False.\",\r\n \"ProcessName\": \"ExcelInvoiceProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0066666-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"InvoiceOps\",\r\n \"FullyQualifiedName\": \"InvoiceOps\",\r\n \"Description\": \"Folder for daily invoice-reporting automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6060001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/task.yaml index f0ed103fa6..959f292edd 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-empty-source/task.yaml @@ -26,21 +26,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/README.md new file mode 100644 index 0000000000..854ae25eab --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/README.md @@ -0,0 +1,60 @@ +# Excel Write Range — Formula-Prefix Character + +This scenario reproduces a Write Range failure where the source +`DataTable` contains a value starting with `=` — interpreted by +Excel COM as a malformed formula. The job ends with: + +``` +System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error. +``` + +## What this scenario uncovers + +**Root Cause:** Row 43 of the scraped HR `DataTable` +(`dtEmployees`) contains `=Smith, John` in the `FullName` column — +a data-entry error in the HR source system. Excel COM's +`Range.set_Value` interprets the leading `=` as a formula prefix, +attempts to parse `=Smith, John` as a formula, fails (unknown name +token `Smith`, comma in the wrong context), and rejects the cell +with the generic 0x800A03EC HRESULT. The workflow has no +sanitization step between the HR scrape and the Excel write, so +the bad data passes through verbatim and the activity surfaces the +rejection as-is. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-range-failures.md` +(branch 5, formula-prefix sub-cause — distinct from branch 5's +volume sub-cause). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelDirectoryProcess` project — `Invoke ScrapeHRDirectory.xaml` → `Use Excel File` → `Write Range` (COM provider) | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (COMException 0x800A03EC +on WriteRange_1, stack through `Range.set_Value`) → `jobs logs` +(the cell-rejection Trace naming `=Smith, John` is the smoking gun) → +workflow source review (confirms no sanitization step between HR +scrape and Write Range) → conclude branch 5 formula-prefix +sub-cause. + +> **Note on volume vs. formula-prefix distinction.** Branch 5 has +> two sub-causes that share the 0x800A03EC signature: oversized +> batches (volume) and formula-prefix data (content). The 312-row +> table is well under the volume threshold; the cell-rejection +> Trace specifically names a formula-prefix value. The test +> rewards agents that distinguish the two sub-causes and pick +> formula-prefix as the correct one. + +> **Note on fixtures.** Synthetic. The HR scrape sub-workflow, row +> count, and exact cell value are placeholders. The test grades +> whether the agent identifies the formula-prefix root + recommends +> a real fix (apostrophe escape, prefix-stripping sanitization, or +> OpenXML provider switch) and avoids the Try-Catch-and-continue +> anti-pattern that would silently truncate the directory. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/20779427df.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/20779427df.json deleted file mode 100644 index 04330c7777..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/20779427df.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-26T08:00:01.300Z", - "EndTime": "2026-05-26T08:00:04.812Z", - "State": "Faulted", - "ReleaseName": "ExcelDirectoryProcess", - "Source": "Schedule", - "Reference": "WeeklyDirectoryA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0077777-3333-4444-5555-666677778888", - "Info": "System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/450e93cc79.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/450e93cc79.json deleted file mode 100644 index 4a748a25dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/450e93cc79.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-26T08:00:01.005Z"}, - {"Key": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-26T08:00:01.300Z"}, - {"Key": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-26T08:00:04.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/7927c35c86.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/7927c35c86.json deleted file mode 100644 index 1d075db473..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/7927c35c86.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0077777-3333-4444-5555-666677778888", - "DisplayName": "DirectoryOps", - "FullyQualifiedName": "DirectoryOps", - "Description": "Folder for HR directory-sync automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6070001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/9a5a316d79.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/9a5a316d79.json deleted file mode 100644 index 00ae8e1796..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/9a5a316d79.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ee3333cc-001", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-26T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-002", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-26T08:00:01.418Z", - "Message": "[ExcelDirectoryProcess] Scraping employee names from HR system and writing directory to C:\\Robot\\Data\\directory.xlsx", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-003", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-26T08:00:02.010Z", - "Message": "Invoke: ScrapeHRDirectory.xaml — returned dtEmployees with 312 rows, 5 columns (Id, FullName, Department, Email, Title)", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-004", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-26T08:00:02.510Z", - "Message": "Use Excel File: directory.xlsx — workbook opened (COM provider; OpenXML fallback declined due to cell-format requirements)", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-005", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-26T08:00:02.810Z", - "Message": "Write Range: employee directory — resolved config Sheet='Directory' StartingCell='A1' ExcludeHeaders=False DataTable='dtEmployees' (rows=312 cols=5)", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-006", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-26T08:00:03.610Z", - "Message": "Write Range: employee directory — marshalling DataTable to Excel COM range A1:E313 (header row + 312 data rows)", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-007", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-26T08:00:04.310Z", - "Message": "Write Range: employee directory — COM Range.set_Value rejected cell B44: source row 43 (0-indexed 42) column 'FullName' has value '=Smith, John' (leading '=' interpreted as malformed formula by Excel)", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee3333cc-008", - "JobKey": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-26T08:00:04.802Z", - "Message": "Write Range: employee directory — System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.", - "ProcessName": "ExcelDirectoryProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/README.md deleted file mode 100644 index f0d97648b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# Excel Write Range — Formula-Prefix Character - -This scenario reproduces a Write Range failure where the source -`DataTable` contains a value starting with `=` — interpreted by -Excel COM as a malformed formula. The job ends with: - -``` -System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error. -``` - -## What this scenario uncovers - -**Root Cause:** Row 43 of the scraped HR `DataTable` -(`dtEmployees`) contains `=Smith, John` in the `FullName` column — -a data-entry error in the HR source system. Excel COM's -`Range.set_Value` interprets the leading `=` as a formula prefix, -attempts to parse `=Smith, John` as a formula, fails (unknown name -token `Smith`, comma in the wrong context), and rejects the cell -with the generic 0x800A03EC HRESULT. The workflow has no -sanitization step between the HR scrape and the Excel write, so -the bad data passes through verbatim and the activity surfaces the -rejection as-is. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-range-failures.md` -(branch 5, formula-prefix sub-cause — distinct from branch 5's -volume sub-cause). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelDirectoryProcess` project — `Invoke ScrapeHRDirectory.xaml` → `Use Excel File` → `Write Range` (COM provider) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture both the resolved Write Range config (rows=312 cols=5) AND the cell-level rejection log naming the exact offending cell value (`=Smith, John` at B44, source row 43, column `FullName`) | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (COMException 0x800A03EC -on WriteRange_1, stack through `Range.set_Value`) → `jobs logs` -(the cell-rejection Trace naming `=Smith, John` is the smoking gun) → -workflow source review (confirms no sanitization step between HR -scrape and Write Range) → conclude branch 5 formula-prefix -sub-cause. - -> **Note on volume vs. formula-prefix distinction.** Branch 5 has -> two sub-causes that share the 0x800A03EC signature: oversized -> batches (volume) and formula-prefix data (content). The 312-row -> table is well under the volume threshold; the cell-rejection -> Trace specifically names a formula-prefix value. The test -> rewards agents that distinguish the two sub-causes and pick -> formula-prefix as the correct one. - -> **Note on fixtures.** Synthetic. The HR scrape sub-workflow, row -> count, and exact cell value are placeholders. The test grades -> whether the agent identifies the formula-prefix root + recommends -> a real fix (apostrophe escape, prefix-stripping sanitization, or -> OpenXML provider switch) and avoids the Try-Catch-and-continue -> anti-pattern that would silently truncate the directory. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/ef9a2bc1c5.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/ef9a2bc1c5.json deleted file mode 100644 index a580034c0c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/ef9a2bc1c5.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-26T08:00:01.300Z", - "EndTime": "2026-05-26T08:00:04.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ee33333-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\r\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\r\n at Microsoft.Office.Interop.Excel.Range.set_Value(Object RangeValueDataType, Object value)\r\n at UiPath.Excel.Activities.Business.WriteRangeX.WriteCellBlock(Range target, DataTable source)\r\n at UiPath.Excel.Activities.Business.WriteRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-26T08:00:01.005Z", - "ReleaseName": "ExcelDirectoryProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680701, - "OrganizationUnitId": 6070001, - "FolderKey": "f0077777-3333-4444-5555-666677778888", - "Reference": "WeeklyDirectoryA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-26T08:00:04.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "COMException", - "ActivityName": "WriteRange_1", - "ActivityDisplayName": "Write Range: employee directory", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ee333333-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680799001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/manifest.json deleted file mode 100644 index 2fd8967795..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Write Range employee-directory scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "7927c35c86.json"}, - {"match": "or folders list --output json", "file": "7927c35c86.json"}, - {"match": "or jobs list --folder-key \"f0077777-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "20779427df.json"}, - {"match": "or jobs list --folder-key f0077777-3333-4444-5555-666677778888 --state Faulted --output json", "file": "20779427df.json"}, - {"match": "or jobs get \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "ef9a2bc1c5.json"}, - {"match": "or jobs get ee333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "ef9a2bc1c5.json"}, - {"match": "or jobs history \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "450e93cc79.json"}, - {"match": "or jobs history ee333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "450e93cc79.json"}, - {"match": "or jobs logs \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "9a5a316d79.json"}, - {"match": "or jobs logs ee333333-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "9a5a316d79.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/uip-fixture.json new file mode 100644 index 0000000000..2a610ffaf3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0077777-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-26T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-26T08:00:04.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelDirectoryProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"WeeklyDirectoryA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0077777-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ee333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-26T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-26T08:00:04.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ee33333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\\r\\n at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)\\r\\n at Microsoft.Office.Interop.Excel.Range.set_Value(Object RangeValueDataType, Object value)\\r\\n at UiPath.Excel.Activities.Business.WriteRangeX.WriteCellBlock(Range target, DataTable source)\\r\\n at UiPath.Excel.Activities.Business.WriteRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-26T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelDirectoryProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680701,\r\n \"OrganizationUnitId\": 6070001,\r\n \"FolderKey\": \"f0077777-3333-4444-5555-666677778888\",\r\n \"Reference\": \"WeeklyDirectoryA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-26T08:00:04.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"COMException\",\r\n \"ActivityName\": \"WriteRange_1\",\r\n \"ActivityDisplayName\": \"Write Range: employee directory\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680799001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ee333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-26T08:00:01.005Z\"},\r\n {\"Key\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-26T08:00:01.300Z\"},\r\n {\"Key\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-26T08:00:04.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ee333333-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ee3333cc-001\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-26T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-002\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-26T08:00:01.418Z\",\r\n \"Message\": \"[ExcelDirectoryProcess] Scraping employee names from HR system and writing directory to C:\\\\Robot\\\\Data\\\\directory.xlsx\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-003\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-26T08:00:02.010Z\",\r\n \"Message\": \"Invoke: ScrapeHRDirectory.xaml — returned dtEmployees with 312 rows, 5 columns (Id, FullName, Department, Email, Title)\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-004\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-26T08:00:02.510Z\",\r\n \"Message\": \"Use Excel File: directory.xlsx — workbook opened (COM provider; OpenXML fallback declined due to cell-format requirements)\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-005\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-26T08:00:02.810Z\",\r\n \"Message\": \"Write Range: employee directory — resolved config Sheet='Directory' StartingCell='A1' ExcludeHeaders=False DataTable='dtEmployees' (rows=312 cols=5)\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-006\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-26T08:00:03.610Z\",\r\n \"Message\": \"Write Range: employee directory — marshalling DataTable to Excel COM range A1:E313 (header row + 312 data rows)\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-007\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-26T08:00:04.310Z\",\r\n \"Message\": \"Write Range: employee directory — COM Range.set_Value rejected cell B44: source row 43 (0-indexed 42) column 'FullName' has value '=Smith, John' (leading '=' interpreted as malformed formula by Excel)\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee3333cc-008\",\r\n \"JobKey\": \"ee333333-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-26T08:00:04.802Z\",\r\n \"Message\": \"Write Range: employee directory — System.Runtime.InteropServices.COMException (0x800A03EC): Application-defined or object-defined error.\",\r\n \"ProcessName\": \"ExcelDirectoryProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0077777-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"DirectoryOps\",\r\n \"FullyQualifiedName\": \"DirectoryOps\",\r\n \"Description\": \"Folder for HR directory-sync automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6070001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/task.yaml index 11258b4aa0..132714c666 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-formula-prefix/task.yaml @@ -28,21 +28,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/README.md new file mode 100644 index 0000000000..b54ff717e9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/README.md @@ -0,0 +1,47 @@ +# Excel Write Range — Uninitialized DataTable + +This scenario reproduces a Write Range failure where the `DataTable` +argument was never assigned because the preceding `Read Range` is +nested inside an `If` activity whose condition evaluated to `False`. +The job ends with: + +``` +System.NullReferenceException: Object reference not set to an instance of an object. +``` + +## What this scenario uncovers + +**Root Cause:** The workflow's Write Range activity receives the +`dtSource` variable, but `dtSource` was never assigned during this +run because the preceding `If File.Exists("C:\Robot\Data\customer-source.xlsx")` +evaluated to `False` and skipped its Then-branch (which contained the +populating `Read Range`). Write Range's argument resolver dereferences +the `Nothing` variable and throws the NRE before any provider call. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-range-failures.md` +(the "Uninitialized / null DataTable" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `ExcelExportProcess` project — `If File.Exists(...)` wrapping the populating `Read Range`, followed by an unconditional `Write Range` against `dtSource` | + +The expected investigation chain: `folders list-current-user` → +`jobs list --state Faulted` → `jobs get` (NullReferenceException on +WriteRange_1) → `jobs logs` (the If-condition-False Trace + the +DataTable-resolved-to-Nothing Trace are the smoking gun) → workflow +source review (confirms the `If` wraps the only assignment to +`dtSource`) → conclude branch 1. + +> **Note on fixtures.** Synthetic. The source workbook path, +> condition expression, and exact NRE stack trace are placeholders. +> The test grades whether the agent connects the skipped-If to the +> never-assigned variable and recommends a real fix (initialize at +> declaration, guard the Write Range, fail-fast on missing source, +> or restructure the workflow to make the dependency explicit). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/528a001c26.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/528a001c26.json deleted file mode 100644 index 50ed24128e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/528a001c26.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-24T08:00:01.300Z", - "EndTime": "2026-05-24T08:00:02.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9ee11111-cccc-dddd-eeee-ffffaaaabbbb", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Excel.Activities.Business.WriteRangeX.ResolveSourceTable(NativeActivityContext context)\r\n at UiPath.Excel.Activities.Business.WriteRangeX.OnExecute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-24T08:00:01.005Z", - "ReleaseName": "ExcelExportProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680501, - "OrganizationUnitId": 6050001, - "FolderKey": "f0055555-3333-4444-5555-666677778888", - "Reference": "NightlyExportA", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-24T08:00:02.812Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "NullReferenceException", - "ActivityName": "WriteRange_1", - "ActivityDisplayName": "Write Range: export customers", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Id": 680599001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/5bc3340905.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/5bc3340905.json deleted file mode 100644 index de3bb7fb04..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/5bc3340905.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "StartTime": "2026-05-24T08:00:01.300Z", - "EndTime": "2026-05-24T08:00:02.812Z", - "State": "Faulted", - "ReleaseName": "ExcelExportProcess", - "Source": "Schedule", - "Reference": "NightlyExportA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0055555-3333-4444-5555-666677778888", - "Info": "System.NullReferenceException: Object reference not set to an instance of an object.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/68ba941af4.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/68ba941af4.json deleted file mode 100644 index 178cff884b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/68ba941af4.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0055555-3333-4444-5555-666677778888", - "DisplayName": "DataExports", - "FullyQualifiedName": "DataExports", - "Description": "Folder for nightly customer-export automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6050001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/96003356e7.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/96003356e7.json deleted file mode 100644 index 91ddf64457..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/96003356e7.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - {"Key": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Pending", "Timestamp": "2026-05-24T08:00:01.005Z"}, - {"Key": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Running", "Timestamp": "2026-05-24T08:00:01.300Z"}, - {"Key": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", "State": "Faulted", "Timestamp": "2026-05-24T08:00:02.812Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/README.md deleted file mode 100644 index 88a693ad81..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Excel Write Range — Uninitialized DataTable - -This scenario reproduces a Write Range failure where the `DataTable` -argument was never assigned because the preceding `Read Range` is -nested inside an `If` activity whose condition evaluated to `False`. -The job ends with: - -``` -System.NullReferenceException: Object reference not set to an instance of an object. -``` - -## What this scenario uncovers - -**Root Cause:** The workflow's Write Range activity receives the -`dtSource` variable, but `dtSource` was never assigned during this -run because the preceding `If File.Exists("C:\Robot\Data\customer-source.xlsx")` -evaluated to `False` and skipped its Then-branch (which contained the -populating `Read Range`). Write Range's argument resolver dereferences -the `Nothing` variable and throws the NRE before any provider call. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/excel-activities/playbooks/write-range-failures.md` -(the "Uninitialized / null DataTable" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `ExcelExportProcess` project — `If File.Exists(...)` wrapping the populating `Read Range`, followed by an unconditional `Write Range` against `dtSource` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the `or jobs logs` Trace entries capture both the `If` condition evaluating to `False` AND the resolver finding `dtSource = Nothing` | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The expected investigation chain: `folders list-current-user` → -`jobs list --state Faulted` → `jobs get` (NullReferenceException on -WriteRange_1) → `jobs logs` (the If-condition-False Trace + the -DataTable-resolved-to-Nothing Trace are the smoking gun) → workflow -source review (confirms the `If` wraps the only assignment to -`dtSource`) → conclude branch 1. - -> **Note on fixtures.** Synthetic. The source workbook path, -> condition expression, and exact NRE stack trace are placeholders. -> The test grades whether the agent connects the skipped-If to the -> never-assigned variable and recommends a real fix (initialize at -> declaration, guard the Write Range, fail-fast on missing source, -> or restructure the workflow to make the dependency explicit). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/da0a0f6f4f.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/da0a0f6f4f.json deleted file mode 100644 index aafad1a3c9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/da0a0f6f4f.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-ee1111cc-001", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-24T08:00:01.310Z", - "Message": "Execution started.", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee1111cc-002", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Info", - "Timestamp": "2026-05-24T08:00:01.418Z", - "Message": "[ExcelExportProcess] Exporting customer data from source to C:\\Robot\\Data\\customer-export.xlsx", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee1111cc-003", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-24T08:00:01.510Z", - "Message": "If source workbook exists — condition: File.Exists(\"C:\\Robot\\Data\\customer-source.xlsx\") evaluated to False (file not found on host filesystem). Then-branch skipped.", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee1111cc-004", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-24T08:00:02.010Z", - "Message": "Use Excel File: customer-export.xlsx — workbook opened (OpenXML provider)", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee1111cc-005", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-24T08:00:02.310Z", - "Message": "Write Range: export customers — resolving DataTable argument 'dtSource'", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee1111cc-006", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Trace", - "Timestamp": "2026-05-24T08:00:02.510Z", - "Message": "Write Range: export customers — DataTable argument 'dtSource' resolved to Nothing (no assignment found in execution path)", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-ee1111cc-007", - "JobKey": "ee111111-cccc-dddd-eeee-ffffaaaabbbb", - "Level": "Error", - "Timestamp": "2026-05-24T08:00:02.802Z", - "Message": "Write Range: export customers — System.NullReferenceException: Object reference not set to an instance of an object.", - "ProcessName": "ExcelExportProcess", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/manifest.json deleted file mode 100644 index 02cd90eae4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Write Range customer-export scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - {"match": "or folders list --output json", "file": "68ba941af4.json"}, - {"match": "or folders list --output json", "file": "68ba941af4.json"}, - {"match": "or jobs list --folder-key \"f0055555-3333-4444-5555-666677778888\" --state Faulted --output json", "file": "5bc3340905.json"}, - {"match": "or jobs list --folder-key f0055555-3333-4444-5555-666677778888 --state Faulted --output json", "file": "5bc3340905.json"}, - {"match": "or jobs get \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "528a001c26.json"}, - {"match": "or jobs get ee111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "528a001c26.json"}, - {"match": "or jobs history \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "96003356e7.json"}, - {"match": "or jobs history ee111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "96003356e7.json"}, - {"match": "or jobs logs \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\" --output json", "file": "da0a0f6f4f.json"}, - {"match": "or jobs logs ee111111-cccc-dddd-eeee-ffffaaaabbbb --output json", "file": "da0a0f6f4f.json"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/uip-fixture.json new file mode 100644 index 0000000000..4d848f3008 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/data/uip-fixture.json @@ -0,0 +1,76 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0055555-3333-4444-5555-666677778888", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-24T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-24T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ExcelExportProcess\",\r\n \"Source\": \"Schedule\",\r\n \"Reference\": \"NightlyExportA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0055555-3333-4444-5555-666677778888\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ee111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"StartTime\": \"2026-05-24T08:00:01.300Z\",\r\n \"EndTime\": \"2026-05-24T08:00:02.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9ee11111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Info\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Excel.Activities.Business.WriteRangeX.ResolveSourceTable(NativeActivityContext context)\\r\\n at UiPath.Excel.Activities.Business.WriteRangeX.OnExecute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-24T08:00:01.005Z\",\r\n \"ReleaseName\": \"ExcelExportProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680501,\r\n \"OrganizationUnitId\": 6050001,\r\n \"FolderKey\": \"f0055555-3333-4444-5555-666677778888\",\r\n \"Reference\": \"NightlyExportA\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-24T08:00:02.812Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"NullReferenceException\",\r\n \"ActivityName\": \"WriteRange_1\",\r\n \"ActivityDisplayName\": \"Write Range: export customers\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Id\": 680599001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "ee111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\"Key\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Pending\", \"Timestamp\": \"2026-05-24T08:00:01.005Z\"},\r\n {\"Key\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Running\", \"Timestamp\": \"2026-05-24T08:00:01.300Z\"},\r\n {\"Key\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\", \"State\": \"Faulted\", \"Timestamp\": \"2026-05-24T08:00:02.812Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ee111111-cccc-dddd-eeee-ffffaaaabbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-ee1111cc-001\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-24T08:00:01.310Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee1111cc-002\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-24T08:00:01.418Z\",\r\n \"Message\": \"[ExcelExportProcess] Exporting customer data from source to C:\\\\Robot\\\\Data\\\\customer-export.xlsx\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee1111cc-003\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-24T08:00:01.510Z\",\r\n \"Message\": \"If source workbook exists — condition: File.Exists(\\\"C:\\\\Robot\\\\Data\\\\customer-source.xlsx\\\") evaluated to False (file not found on host filesystem). Then-branch skipped.\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee1111cc-004\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-24T08:00:02.010Z\",\r\n \"Message\": \"Use Excel File: customer-export.xlsx — workbook opened (OpenXML provider)\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee1111cc-005\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-24T08:00:02.310Z\",\r\n \"Message\": \"Write Range: export customers — resolving DataTable argument 'dtSource'\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee1111cc-006\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-24T08:00:02.510Z\",\r\n \"Message\": \"Write Range: export customers — DataTable argument 'dtSource' resolved to Nothing (no assignment found in execution path)\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-ee1111cc-007\",\r\n \"JobKey\": \"ee111111-cccc-dddd-eeee-ffffaaaabbbb\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-24T08:00:02.802Z\",\r\n \"Message\": \"Write Range: export customers — System.NullReferenceException: Object reference not set to an instance of an object.\",\r\n \"ProcessName\": \"ExcelExportProcess\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0055555-3333-4444-5555-666677778888\",\r\n \"DisplayName\": \"DataExports\",\r\n \"FullyQualifiedName\": \"DataExports\",\r\n \"Description\": \"Folder for nightly customer-export automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6050001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/task.yaml index 910e8d777a..f6e2af46fc 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/excel-write-range-uninitialized-datatable/task.yaml @@ -28,21 +28,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/README.md new file mode 100644 index 0000000000..43bf9cca16 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/README.md @@ -0,0 +1,45 @@ +# GSuite Add-Sheet Name-Conflict Troubleshooting — Faithful Replay + +This scenario replays a real `uipath-troubleshoot` investigation of a Faulted +Orchestrator job whose UiPath.GSuite **Add Sheet** activity hit a duplicate +**sheet-name conflict**. + +## What the original session uncovered + +The user asked only *"why did my last job from folder Shared fail?"*. The agent: + +1. Listed folders → resolved **Shared** to key `1965a46b-db4e-469e-aaaa-7e0b379cb34d`. +2. Listed Faulted jobs in Shared → newest is **SheetTabBuilder**, + job `2fbf6ba4-b347-462f-a83e-33c6c34bbea6`. +3. Read the job's `Info` → a `UiPath.GSuite.Exceptions.GSuiteException` thrown + from `AddSheetWithConflictResolution` in `AddSheetConnections`. + +Conclusion: the target spreadsheet already had a tab matching the configured +`SheetName` and `ConflictResolution` was left at `Fail`, so Add Sheet threw a +same-name conflict — *not* a connection, auth, 404, invalid-range, throttling, +or transient fault. (Listing connections confirms both GSuite connections +serving Shared are Enabled and valid — a defensive fixture reinforces that the +connection is fine.) + +## Name-conflict signature + +``` +UiPath.GSuite.Exceptions.GSuiteException: A sheet with the same name already exists. + at UiPath.GSuite.Sheets.Extensions.ISheetsServiceExtensions.AddSheetWithConflictResolution(...) +``` + +## How this test reproduces it + +| Layer | Source | +|---|---| + +CLI-driven — no `process/` snapshot; the root cause is fully contained in the +`or jobs get` `Info` field. + +## Playbook validated + +`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/add-sheet-name-conflict.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/0bc585a8d2.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/0bc585a8d2.json deleted file mode 100644 index 6bcd23655d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/0bc585a8d2.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-12T15:34:08.683Z", - "JobId": 4015143 - }, - { - "State": "Running", - "Timestamp": "2026-06-12T15:34:04.767Z", - "JobId": 4015143 - }, - { - "State": "Pending", - "Timestamp": "2026-06-12T15:34:03.773Z", - "JobId": 4015143 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/0dbd70a0de.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/0dbd70a0de.json deleted file mode 100644 index 9e46f0e7dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/0dbd70a0de.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "StartTime": "2026-06-12T15:34:04.717Z", - "EndTime": "2026-06-12T15:34:08.643Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "8aea37c9-6094-4f4d-9a8f-4b236ccf02d2", - "Info": "A sheet with the same name already exists.\r\n\r\nin Main.xaml\r\n at AddSheetConnections \"Add Sheet\"\r\n at Sequence \"Add Sheet\"\r\n at Main \"Main\"\r\n\r\nUiPath.GSuite.Exceptions.GSuiteException: A sheet with the same name already exists.\r\n at UiPath.GSuite.Sheets.Extensions.ISheetsServiceExtensions.AddSheetWithConflictResolution(ISheetsService sheetsService, AddNewSheetRequest addNewSheetRequest, ConflictBehaviorMaster conflictResolution, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.AddSheetConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-12T15:34:03.737Z", - "ReleaseName": "SheetTabBuilder", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53482, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T15:34:08.643Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "2fbf6ba4b347462fa83e33c6c34bbea6", - "Id": 4015143 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/262f7601dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/262f7601dc.json deleted file mode 100644 index 441b6a0c42..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/262f7601dc.json +++ /dev/null @@ -1,432 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "State": "Faulted", - "ProcessName": "SheetTabBuilder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T15:34:04.717Z", - "EndTime": "2026-06-12T15:34:08.643Z", - "CreationTime": "2026-06-12T15:34:03.737Z" - }, - { - "Key": "d5e0e6b0-55b8-4256-8494-c601cb986dab", - "State": "Faulted", - "ProcessName": "SheetRangeWriter", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T15:11:29.767Z", - "EndTime": "2026-06-12T15:11:34.253Z", - "CreationTime": "2026-06-12T15:11:29.410Z" - }, - { - "Key": "4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "State": "Faulted", - "ProcessName": "WorkspaceSheetSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T14:38:50.020Z", - "EndTime": "2026-06-12T14:38:56.633Z", - "CreationTime": "2026-06-12T14:38:49.027Z" - }, - { - "Key": "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "State": "Faulted", - "ProcessName": "SheetExportRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:59:41.830Z", - "EndTime": "2026-06-12T13:59:48.663Z", - "CreationTime": "2026-06-12T13:59:41.113Z" - }, - { - "Key": "f41ed418-163b-487b-8077-e51f8af4a1af", - "State": "Faulted", - "ProcessName": "DriveDocSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:21:01.380Z", - "EndTime": "2026-06-12T13:21:12.450Z", - "CreationTime": "2026-06-12T13:21:00.577Z" - }, - { - "Key": "5242daf1-a4e9-41d2-b828-3b70e9d59cc1", - "State": "Faulted", - "ProcessName": "InboxSearchBot", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T09:49:32.447Z", - "EndTime": "2026-06-11T09:49:41.220Z", - "CreationTime": "2026-06-11T09:49:31.683Z" - }, - { - "Key": "7fd5e890-f162-4591-a895-e649b09aea02", - "State": "Faulted", - "ProcessName": "DocumentUploader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:15:47.690Z", - "EndTime": "2026-06-11T08:15:54.983Z", - "CreationTime": "2026-06-11T08:15:46.967Z" - }, - { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "State": "Faulted", - "ProcessName": "TeamMailRetriever", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "CreationTime": "2026-06-11T08:01:50.607Z" - }, - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 30, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/790c63a18f.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/790c63a18f.json deleted file mode 100644 index 17128079bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/790c63a18f.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "510ef231-2b59-4468-8808-141c9e6ce8bf", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-drive", - "ConnectorName": "Google Drive", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714201, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:14:22.110Z", - "Updated": "2026-06-10T11:48:03.921Z" - }, - { - "Id": "b03d73b7-611a-4e0e-b109-a85c10be40c8", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-sheets", - "ConnectorName": "Google Sheets", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714202, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:15:01.554Z", - "Updated": "2026-06-10T11:48:31.207Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/README.md deleted file mode 100644 index 230885f1d7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# GSuite Add-Sheet Name-Conflict Troubleshooting — Faithful Replay - -This scenario replays a real `uipath-troubleshoot` investigation of a Faulted -Orchestrator job whose UiPath.GSuite **Add Sheet** activity hit a duplicate -**sheet-name conflict**. - -## What the original session uncovered - -The user asked only *"why did my last job from folder Shared fail?"*. The agent: - -1. Listed folders → resolved **Shared** to key `1965a46b-db4e-469e-aaaa-7e0b379cb34d`. -2. Listed Faulted jobs in Shared → newest is **SheetTabBuilder**, - job `2fbf6ba4-b347-462f-a83e-33c6c34bbea6`. -3. Read the job's `Info` → a `UiPath.GSuite.Exceptions.GSuiteException` thrown - from `AddSheetWithConflictResolution` in `AddSheetConnections`. - -Conclusion: the target spreadsheet already had a tab matching the configured -`SheetName` and `ConflictResolution` was left at `Fail`, so Add Sheet threw a -same-name conflict — *not* a connection, auth, 404, invalid-range, throttling, -or transient fault. (Listing connections confirms both GSuite connections -serving Shared are Enabled and valid — a defensive fixture reinforces that the -connection is fine.) - -## Name-conflict signature - -``` -UiPath.GSuite.Exceptions.GSuiteException: A sheet with the same name already exists. - at UiPath.GSuite.Sheets.Extensions.ISheetsServiceExtensions.AddSheetWithConflictResolution(...) -``` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven dispatcher) | -| `fixtures/mocks/responses/*.json` | verbatim, scrubbed stdout from the real session's `.local/investigations/raw/` | -| `fixtures/mocks/responses/manifest.json` | first-match-wins dispatch table; `docsai ask` is passthrough | - -CLI-driven — no `process/` snapshot; the root cause is fully contained in the -`or jobs get` `Info` field. - -## Playbook validated - -`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/add-sheet-name-conflict.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/f6b9fc4e33.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/f6b9fc4e33.json deleted file mode 100644 index 33cab91370..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/m/r/f6b9fc4e33.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error reading job traces response", - "Instructions": "Unexpected token '<', \" MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving the GSuite connections are valid/Enabled (the fault is NOT a connection problem). `or jobs traces` returns the real non-JSON Failure envelope the live CLI produced.", - "rules": [ - { - "match": "or jobs get 2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "file": "0dbd70a0de.json", - "exit_code": 0 - }, - { - "match": "or jobs history 2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "file": "0bc585a8d2.json", - "exit_code": 0 - }, - { - "match": "or jobs logs 2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "file": "fc3fc04134.json", - "exit_code": 0 - }, - { - "match": "or jobs traces 2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "262f7601dc.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "262f7601dc.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "262f7601dc.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (2fbf6ba4 is the latest by StartTime)." - }, - { - "pattern": "or jobs get 2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the GSuiteException / same-name-conflict stack from AddSheetWithConflictResolution." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL — Agent may list connections to rule out a connection/auth problem; the fixture shows both GSuite connections Enabled. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces 2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "min": 0, - "description": "OPTIONAL — traces are unavailable for this job (non-JSON Failure envelope); tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/uip-fixture.json new file mode 100644 index 0000000000..132d6e81b1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-add-sheet-name-conflict/data/uip-fixture.json @@ -0,0 +1,136 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2fbf6ba4-b347-462f-a83e-33c6c34bbea6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetTabBuilder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:34:04.717Z\",\r\n \"EndTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"CreationTime\": \"2026-06-12T15:34:03.737Z\"\r\n },\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 30,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2fbf6ba4-b347-462f-a83e-33c6c34bbea6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetTabBuilder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:34:04.717Z\",\r\n \"EndTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"CreationTime\": \"2026-06-12T15:34:03.737Z\"\r\n },\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 30,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "2fbf6ba4-b347-462f-a83e-33c6c34bbea6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"2fbf6ba4-b347-462f-a83e-33c6c34bbea6\",\r\n \"StartTime\": \"2026-06-12T15:34:04.717Z\",\r\n \"EndTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"8aea37c9-6094-4f4d-9a8f-4b236ccf02d2\",\r\n \"Info\": \"A sheet with the same name already exists.\\r\\n\\r\\nin Main.xaml\\r\\n at AddSheetConnections \\\"Add Sheet\\\"\\r\\n at Sequence \\\"Add Sheet\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.GSuite.Exceptions.GSuiteException: A sheet with the same name already exists.\\r\\n at UiPath.GSuite.Sheets.Extensions.ISheetsServiceExtensions.AddSheetWithConflictResolution(ISheetsService sheetsService, AddNewSheetRequest addNewSheetRequest, ConflictBehaviorMaster conflictResolution, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.AddSheetConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T15:34:03.737Z\",\r\n \"ReleaseName\": \"SheetTabBuilder\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53482,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"2fbf6ba4b347462fa83e33c6c34bbea6\",\r\n \"Id\": 4015143\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "2fbf6ba4-b347-462f-a83e-33c6c34bbea6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T15:34:08.683Z\",\r\n \"JobId\": 4015143\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T15:34:04.767Z\",\r\n \"JobId\": 4015143\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-12T15:34:03.773Z\",\r\n \"JobId\": 4015143\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2fbf6ba4-b347-462f-a83e-33c6c34bbea6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T15:34:08.321Z\",\r\n \"Message\": \"Add Sheet: A sheet with the same name already exists.\\r\\nin Main.xaml\\r\\n at AddSheetConnections \\\"Add Sheet\\\"\\r\\n at Sequence \\\"Add Sheet\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.GSuite.Exceptions.GSuiteException\\r\\n at UiPath.GSuite.Sheets.Extensions.ISheetsServiceExtensions.AddSheetWithConflictResolution(ISheetsService sheetsService, AddNewSheetRequest addNewSheetRequest, ConflictBehaviorMaster conflictResolution, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.AddSheetConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "2fbf6ba4-b347-462f-a83e-33c6c34bbea6" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\" UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it + at UiPath.ConnectionClient.ConnectionClient.GetAccessToken(...) +``` + +## How this test reproduces it + +| Layer | Source | +|---|---| + +CLI-driven — no `process/` snapshot; the root cause is fully contained in the +`or jobs get` `Info` field. + +## Playbook validated + +`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/connection-and-auth-failures.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/114660ee1e.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/114660ee1e.json deleted file mode 100644 index ee4afd2e36..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/114660ee1e.json +++ /dev/null @@ -1,404 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "State": "Faulted", - "ProcessName": "WorkspaceSheetSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T14:38:50.020Z", - "EndTime": "2026-06-12T14:38:56.633Z", - "CreationTime": "2026-06-12T14:38:49.027Z" - }, - { - "Key": "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "State": "Faulted", - "ProcessName": "SheetExportRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:59:41.830Z", - "EndTime": "2026-06-12T13:59:48.663Z", - "CreationTime": "2026-06-12T13:59:41.113Z" - }, - { - "Key": "f41ed418-163b-487b-8077-e51f8af4a1af", - "State": "Faulted", - "ProcessName": "DriveDocSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:21:01.380Z", - "EndTime": "2026-06-12T13:21:12.450Z", - "CreationTime": "2026-06-12T13:21:00.577Z" - }, - { - "Key": "5242daf1-a4e9-41d2-b828-3b70e9d59cc1", - "State": "Faulted", - "ProcessName": "InboxSearchBot", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T09:49:32.447Z", - "EndTime": "2026-06-11T09:49:41.220Z", - "CreationTime": "2026-06-11T09:49:31.683Z" - }, - { - "Key": "7fd5e890-f162-4591-a895-e649b09aea02", - "State": "Faulted", - "ProcessName": "DocumentUploader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:15:47.690Z", - "EndTime": "2026-06-11T08:15:54.983Z", - "CreationTime": "2026-06-11T08:15:46.967Z" - }, - { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "State": "Faulted", - "ProcessName": "TeamMailRetriever", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "CreationTime": "2026-06-11T08:01:50.607Z" - }, - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 28, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/2d5152b547.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/2d5152b547.json deleted file mode 100644 index 4dbd16aaf9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/2d5152b547.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "StartTime": "2026-06-12T14:38:50.020Z", - "EndTime": "2026-06-12T14:38:56.633Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f46a058a-284f-4687-a63c-ac6d24961e58", - "Info": "Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\r\n\r\nin Main.xaml\r\n at DownloadFileConnections \"Download File\"\r\n at Sequence \"Download File\"\r\n at Main \"Main\"\r\n\r\nUiPath.GSuite.Exceptions.GSuiteException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\r\n ---> UiPath.GSuite.Exceptions.GSuiteInternalException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\r\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\r\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass8_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.ConnectionClient.GetAccessToken(String connectionId, Boolean forceRefresh, CancellationToken ct)\r\n at UiPath.GSuite.Services.GoogleConnectionClient.GetTokenInternalAsync(String connectionId, GetTokenOption forceRefresh, CancellationToken ct)\r\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.GetTokenAsync(CancellationToken cancellationToken)\r\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis.Core\\Http\\ConfigurableMessageHandler.cs:line 446\r\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteUnparsedAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 244\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 214\r\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass8_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.ConnectionClient.GetAccessToken(String connectionId, Boolean forceRefresh, CancellationToken ct)\r\n at UiPath.GSuite.Services.GoogleConnectionClient.GetTokenInternalAsync(String connectionId, GetTokenOption forceRefresh, CancellationToken ct)\r\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.GetTokenAsync(CancellationToken cancellationToken)\r\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis.Core\\Http\\ConfigurableMessageHandler.cs:line 446\r\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteUnparsedAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 244\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 214\r\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-12T14:38:49.027Z", - "ReleaseName": "WorkspaceSheetSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53480, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T14:38:56.633Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "4e00b4cab35b4457bb1274f270d20cb0", - "Id": 4015140 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/790c63a18f.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/790c63a18f.json deleted file mode 100644 index 17128079bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/790c63a18f.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "510ef231-2b59-4468-8808-141c9e6ce8bf", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-drive", - "ConnectorName": "Google Drive", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714201, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:14:22.110Z", - "Updated": "2026-06-10T11:48:03.921Z" - }, - { - "Id": "b03d73b7-611a-4e0e-b109-a85c10be40c8", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-sheets", - "ConnectorName": "Google Sheets", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714202, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:15:01.554Z", - "Updated": "2026-06-10T11:48:31.207Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/84a5033ae2.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/84a5033ae2.json deleted file mode 100644 index c2b11d02b5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/84a5033ae2.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-12T14:38:56.277Z", - "Message": "Download File: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\r\nin Main.xaml\r\n at DownloadFileConnections \"Download File\"\r\n at Sequence \"Download File\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.GSuite.Exceptions.GSuiteException\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/README.md deleted file mode 100644 index cba9b49070..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# GSuite Connection-Invalid Troubleshooting — Faithful Replay - -This scenario replays a real `uipath-troubleshoot` investigation of a Faulted -Orchestrator job whose UiPath.GSuite **Download File** activity referenced an -Integration Service connection that could not be resolved or authorized. - -## What the original session uncovered - -The user asked only *"why did my last job from folder Shared fail?"*. The agent: - -1. Listed folders → resolved **Shared** to key `1965a46b-db4e-469e-aaaa-7e0b379cb34d`. -2. Listed Faulted jobs in Shared → newest is **WorkspaceSheetSync**, - job `4e00b4ca-b35b-4457-bb12-74f270d20cb0`. -3. Read the job's `Info` → a `UiPath.GSuite.Exceptions.GSuiteException` - wrapping `ConnectionHttpException`, thrown from `GetAccessToken` at OAuth - token acquisition — **before** any Google Drive call. - -Conclusion: an Integration Service **connection-resolution / authorization** -failure — the referenced connection `00000000-1111-2222-3333-444444444444` is -invalid or not accessible to the running identity — *not* a Drive 404, bad -input, throttling, transient, or timeout fault. Listing connections corroborates -this: both real GSuite connections serving Shared are Enabled, and the bogus id -the activity references is not among them. - -## Connection-invalid signature - -``` -UiPath.GSuite.Exceptions.GSuiteException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it - ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it - at UiPath.ConnectionClient.ConnectionClient.GetAccessToken(...) -``` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven dispatcher) | -| `fixtures/mocks/responses/*.json` | verbatim, scrubbed stdout from the real session's `.local/investigations/raw/` | -| `fixtures/mocks/responses/manifest.json` | first-match-wins dispatch table; `docsai ask` is passthrough | - -CLI-driven — no `process/` snapshot; the root cause is fully contained in the -`or jobs get` `Info` field. - -## Playbook validated - -`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/connection-and-auth-failures.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/e7e6cf519e.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/e7e6cf519e.json deleted file mode 100644 index f69471ca9b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/e7e6cf519e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-12T14:38:56.703Z", - "JobId": 4015140 - }, - { - "State": "Running", - "Timestamp": "2026-06-12T14:38:50.067Z", - "JobId": 4015140 - }, - { - "State": "Pending", - "Timestamp": "2026-06-12T14:38:49.077Z", - "JobId": 4015140 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/f6b9fc4e33.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/f6b9fc4e33.json deleted file mode 100644 index 33cab91370..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/m/r/f6b9fc4e33.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error reading job traces response", - "Instructions": "Unexpected token '<', \" MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving both GSuite connections serving Shared are valid/Enabled; the bogus connection 00000000-... is intentionally NOT in that list, corroborating that the referenced connection is not resolvable. `or jobs traces` returns the real non-JSON Failure envelope the live CLI produced.", - "rules": [ - { - "match": "or jobs get 4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "file": "2d5152b547.json", - "exit_code": 0 - }, - { - "match": "or jobs history 4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "file": "e7e6cf519e.json", - "exit_code": 0 - }, - { - "match": "or jobs logs 4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "file": "84a5033ae2.json", - "exit_code": 0 - }, - { - "match": "or jobs traces 4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "114660ee1e.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "114660ee1e.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "114660ee1e.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (4e00b4ca is the latest by StartTime)." - }, - { - "pattern": "or jobs get 4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the GSuiteException / ConnectionHttpException stack and the bogus connection id." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL — Agent may list connections to confirm the bogus connection id (00000000-...) is not present and both real GSuite connections are Enabled. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces 4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "min": 0, - "description": "OPTIONAL — traces are unavailable for this job (non-JSON Failure envelope); tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/uip-fixture.json new file mode 100644 index 0000000000..8b88115efd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-connection-invalid/data/uip-fixture.json @@ -0,0 +1,136 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 28,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 28,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "4e00b4ca-b35b-4457-bb12-74f270d20cb0" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f46a058a-284f-4687-a63c-ac6d24961e58\",\r\n \"Info\": \"Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFileConnections \\\"Download File\\\"\\r\\n at Sequence \\\"Download File\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.GSuite.Exceptions.GSuiteException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\\r\\n ---> UiPath.GSuite.Exceptions.GSuiteInternalException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\\r\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass8_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetAccessToken(String connectionId, Boolean forceRefresh, CancellationToken ct)\\r\\n at UiPath.GSuite.Services.GoogleConnectionClient.GetTokenInternalAsync(String connectionId, GetTokenOption forceRefresh, CancellationToken ct)\\r\\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.GetTokenAsync(CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)\\r\\n at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis.Core\\\\Http\\\\ConfigurableMessageHandler.cs:line 446\\r\\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteUnparsedAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 244\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 214\\r\\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass8_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetAccessToken(String connectionId, Boolean forceRefresh, CancellationToken ct)\\r\\n at UiPath.GSuite.Services.GoogleConnectionClient.GetTokenInternalAsync(String connectionId, GetTokenOption forceRefresh, CancellationToken ct)\\r\\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.GetTokenAsync(CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Authentication.Google.OAuth2.ConnectionCredential.InterceptAsync(HttpRequestMessage request, CancellationToken cancellationToken)\\r\\n at Google.Apis.Http.ConfigurableMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis.Core\\\\Http\\\\ConfigurableMessageHandler.cs:line 446\\r\\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteUnparsedAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 244\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 214\\r\\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\",\r\n \"ReleaseName\": \"WorkspaceSheetSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53480,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"4e00b4cab35b4457bb1274f270d20cb0\",\r\n \"Id\": 4015140\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "4e00b4ca-b35b-4457-bb12-74f270d20cb0" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T14:38:56.703Z\",\r\n \"JobId\": 4015140\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T14:38:50.067Z\",\r\n \"JobId\": 4015140\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-12T14:38:49.077Z\",\r\n \"JobId\": 4015140\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4e00b4ca-b35b-4457-bb12-74f270d20cb0" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T14:38:56.277Z\",\r\n \"Message\": \"Download File: Connection [00000000-1111-2222-3333-444444444444] is invalid or you do not have access to it\\r\\nin Main.xaml\\r\\n at DownloadFileConnections \\\"Download File\\\"\\r\\n at Sequence \\\"Download File\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.GSuite.Exceptions.GSuiteException\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "4e00b4ca-b35b-4457-bb12-74f270d20cb0" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\" UiPath.GSuite.Exceptions.GSuiteInternalException: The resource was not found.\r\n ---> Google.GoogleApiException: The service drive has thrown an exception. HttpStatusCode is NotFound. File not found: 1AbCnonexistentFILEid000000000000000.\r\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 256\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 217\r\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 256\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 217\r\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-12T13:21:00.577Z", - "ReleaseName": "DriveDocSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53478, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T13:21:12.450Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f41ed418163b487b8077e51f8af4a1af", - "Id": 4015138 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/790c63a18f.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/790c63a18f.json deleted file mode 100644 index 17128079bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/790c63a18f.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "510ef231-2b59-4468-8808-141c9e6ce8bf", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-drive", - "ConnectorName": "Google Drive", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714201, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:14:22.110Z", - "Updated": "2026-06-10T11:48:03.921Z" - }, - { - "Id": "b03d73b7-611a-4e0e-b109-a85c10be40c8", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-sheets", - "ConnectorName": "Google Sheets", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714202, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:15:01.554Z", - "Updated": "2026-06-10T11:48:31.207Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/README.md deleted file mode 100644 index 9a34601aa7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# GSuite Drive File-Not-Found Troubleshooting — Faithful Replay - -This scenario replays a real `uipath-troubleshoot` investigation of a Faulted -Orchestrator job whose UiPath.GSuite **Download File** activity hit a Google -Drive **404**. - -## What the original session uncovered - -The user asked only *"why did my last job from folder Shared fail?"*. The agent: - -1. Listed folders → resolved **Shared** to key `1965a46b-db4e-469e-aaaa-7e0b379cb34d`. -2. Listed Faulted jobs in Shared → newest is **DriveDocSync**, - job `f41ed418-163b-487b-8077-e51f8af4a1af`. -3. Read the job's `Info` → a `UiPath.GSuite.Exceptions.GSuiteException` - wrapping `Google.GoogleApiException` with **HttpStatusCode NotFound** and - `File not found: 1AbCnonexistentFILEid000000000000000.` thrown from - `DownloadFileConnections`. - -Conclusion: a Google Drive **404 / resource-resolution** failure on the -configured file id — *not* a connection, auth, scope, throttling, or transient -fault. (Listing connections confirms both GSuite connections serving Shared are -Enabled and valid — a defensive fixture reinforces that the connection is fine.) - -## 404 signature - -``` -Google.GoogleApiException: The service drive has thrown an exception. -HttpStatusCode is NotFound. File not found: 1AbCnonexistentFILEid000000000000000. -``` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven dispatcher) | -| `fixtures/mocks/responses/*.json` | verbatim, scrubbed stdout from the real session's `.local/investigations/raw/` | -| `fixtures/mocks/responses/manifest.json` | first-match-wins dispatch table; `docsai ask` is passthrough | - -CLI-driven — no `process/` snapshot; the root cause is fully contained in the -`or jobs get` `Info` field. - -## Playbook validated - -`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/drive-file-not-found.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/e97230ab78.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/e97230ab78.json deleted file mode 100644 index dfc6588d83..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/e97230ab78.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-12T13:21:11.882Z", - "Message": "Download File: The resource was not found.\r\nin Main.xaml\r\n at DownloadFileConnections \"Download File\"\r\n at Sequence \"Download File\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.GSuite.Exceptions.GSuiteException\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/f6b9fc4e33.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/f6b9fc4e33.json deleted file mode 100644 index 33cab91370..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/m/r/f6b9fc4e33.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error reading job traces response", - "Instructions": "Unexpected token '<', \" MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving the GSuite connections are valid/Enabled (the fault is NOT a connection problem). `or jobs traces` returns the real non-JSON Failure envelope the live CLI produced.", - "rules": [ - { - "match": "or jobs get f41ed418-163b-487b-8077-e51f8af4a1af", - "file": "72fd7ab62a.json", - "exit_code": 0 - }, - { - "match": "or jobs history f41ed418-163b-487b-8077-e51f8af4a1af", - "file": "38090516f3.json", - "exit_code": 0 - }, - { - "match": "or jobs logs f41ed418-163b-487b-8077-e51f8af4a1af", - "file": "e97230ab78.json", - "exit_code": 0 - }, - { - "match": "or jobs traces f41ed418-163b-487b-8077-e51f8af4a1af", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "24807e152d.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "24807e152d.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "24807e152d.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (f41ed418 is the latest by StartTime)." - }, - { - "pattern": "or jobs get f41ed418-163b-487b-8077-e51f8af4a1af", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the GSuiteException / Google 404 stack and the bogus file id." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL — Agent may list connections to rule out a connection/auth problem; the fixture shows both GSuite connections Enabled. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces f41ed418-163b-487b-8077-e51f8af4a1af", - "min": 0, - "description": "OPTIONAL — traces are unavailable for this job (non-JSON Failure envelope); tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..7a337e6ac0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-file-not-found/data/uip-fixture.json @@ -0,0 +1,136 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 26,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 26,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f41ed418-163b-487b-8077-e51f8af4a1af" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ceededa5-7b27-422c-ac51-14eb5f3cf7c2\",\r\n \"Info\": \"The resource was not found.\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFileConnections \\\"Download File\\\"\\r\\n at Sequence \\\"Download File\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.GSuite.Exceptions.GSuiteException: The resource was not found.\\r\\n ---> UiPath.GSuite.Exceptions.GSuiteInternalException: The resource was not found.\\r\\n ---> Google.GoogleApiException: The service drive has thrown an exception. HttpStatusCode is NotFound. File not found: 1AbCnonexistentFILEid000000000000000.\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 256\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 217\\r\\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 256\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 217\\r\\n at UiPath.GSuite.Drive.Extensions.DriveExtensions.GetFileAsync(DriveService client, CloudObjectIdentifier identifier, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.DownloadOrExportLocallyAsync(DownloadAndExportRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.DownloadFileConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\",\r\n \"ReleaseName\": \"DriveDocSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53478,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f41ed418163b487b8077e51f8af4a1af\",\r\n \"Id\": 4015138\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "f41ed418-163b-487b-8077-e51f8af4a1af" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T13:21:12.467Z\",\r\n \"JobId\": 4015138\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T13:21:01.437Z\",\r\n \"JobId\": 4015138\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-12T13:21:00.613Z\",\r\n \"JobId\": 4015138\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f41ed418-163b-487b-8077-e51f8af4a1af" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T13:21:11.882Z\",\r\n \"Message\": \"Download File: The resource was not found.\\r\\nin Main.xaml\\r\\n at DownloadFileConnections \\\"Download File\\\"\\r\\n at Sequence \\\"Download File\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.GSuite.Exceptions.GSuiteException\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "f41ed418-163b-487b-8077-e51f8af4a1af" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\" MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving both GSuite connections serving Shared are valid/Enabled (the fault is NOT a connection problem). `or jobs traces` returns the real non-JSON Failure envelope the live CLI produced.", - "rules": [ - { - "match": "or jobs get 6c8b1949-2603-437d-86a6-6b82b989fd6a", - "file": "88a80668f4.json", - "exit_code": 0 - }, - { - "match": "or jobs history 6c8b1949-2603-437d-86a6-6b82b989fd6a", - "file": "49d7ebc24f.json", - "exit_code": 0 - }, - { - "match": "or jobs logs 6c8b1949-2603-437d-86a6-6b82b989fd6a", - "file": "dc1d4b643f.json", - "exit_code": 0 - }, - { - "match": "or jobs traces 6c8b1949-2603-437d-86a6-6b82b989fd6a", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "fc95465511.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "fc95465511.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "fc95465511.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (6c8b1949 is the latest by StartTime)." - }, - { - "pattern": "or jobs get 6c8b1949-2603-437d-86a6-6b82b989fd6a", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the GSuiteException name-conflict (`Multiple items with the name dupe-folder found in the specified folder.`) thrown from CreateFolderConnections." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL -- Agent may list connections to rule out a connection/auth problem; the fixture shows both GSuite connections Enabled. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces 6c8b1949-2603-437d-86a6-6b82b989fd6a", - "min": 0, - "description": "OPTIONAL -- traces are unavailable for this job (non-JSON Failure envelope); tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-multiple-items-name-conflict/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-multiple-items-name-conflict/data/uip-fixture.json new file mode 100644 index 0000000000..658ab6d83d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-drive-multiple-items-name-conflict/data/uip-fixture.json @@ -0,0 +1,136 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6c8b1949-2603-437d-86a6-6b82b989fd6a\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFileCreator\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:58:40.333Z\",\r\n \"EndTime\": \"2026-06-12T15:58:44.587Z\",\r\n \"CreationTime\": \"2026-06-12T15:58:39.873Z\"\r\n },\r\n {\r\n \"Key\": \"2fbf6ba4-b347-462f-a83e-33c6c34bbea6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetTabBuilder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:34:04.717Z\",\r\n \"EndTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"CreationTime\": \"2026-06-12T15:34:03.737Z\"\r\n },\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 31,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6c8b1949-2603-437d-86a6-6b82b989fd6a\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFileCreator\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:58:40.333Z\",\r\n \"EndTime\": \"2026-06-12T15:58:44.587Z\",\r\n \"CreationTime\": \"2026-06-12T15:58:39.873Z\"\r\n },\r\n {\r\n \"Key\": \"2fbf6ba4-b347-462f-a83e-33c6c34bbea6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetTabBuilder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:34:04.717Z\",\r\n \"EndTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"CreationTime\": \"2026-06-12T15:34:03.737Z\"\r\n },\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 31,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "6c8b1949-2603-437d-86a6-6b82b989fd6a" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"6c8b1949-2603-437d-86a6-6b82b989fd6a\",\r\n \"StartTime\": \"2026-06-12T15:58:40.333Z\",\r\n \"EndTime\": \"2026-06-12T15:58:44.587Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"309e85b9-9c9d-48c4-a823-13d835b70c2c\",\r\n \"Info\": \"Multiple items with the name dupe-folder found in the specified folder.\\r\\n\\r\\nin Main.xaml\\r\\n at CreateFolderConnections \\\"Create Folder\\\"\\r\\n at Sequence \\\"Create Folder\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.GSuite.Exceptions.GSuiteException: Multiple items with the name dupe-folder found in the specified folder.\\r\\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.CreateFolderAsync(CreateFolderRequest createFolderRequest, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.CreateFolderConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T15:58:39.873Z\",\r\n \"ReleaseName\": \"DriveFileCreator\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53483,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T15:58:44.587Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"6c8b19492603437d86a66b82b989fd6a\",\r\n \"Id\": 4015144\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "6c8b1949-2603-437d-86a6-6b82b989fd6a" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T15:58:44.593Z\",\r\n \"JobId\": 4015144\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T15:58:40.340Z\",\r\n \"JobId\": 4015144\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-12T15:58:39.880Z\",\r\n \"JobId\": 4015144\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "6c8b1949-2603-437d-86a6-6b82b989fd6a" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T15:58:44.238Z\",\r\n \"Message\": \"Create Folder: Multiple items with the name dupe-folder found in the specified folder.\\r\\nin Main.xaml\\r\\n at CreateFolderConnections \\\"Create Folder\\\"\\r\\n at Sequence \\\"Create Folder\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.GSuite.Exceptions.GSuiteException\\r\\n at UiPath.GSuite.Drive.Services.DriveServiceProxy.CreateFolderAsync(CreateFolderRequest createFolderRequest, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.CreateFolderConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "6c8b1949-2603-437d-86a6-6b82b989fd6a" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\"` segment), so the activity threw *before* any request reached Google +Drive (no HTTP status, no API round-trip). *Not* a Google API, 404, connection, +auth, scope, throttling, outage, or transient fault. (Listing connections +confirms both GSuite connections serving Shared are Enabled and valid — a +defensive fixture reinforces that the connection is fine.) + +## Client-side malformed-URL signature + +``` +System.ArgumentOutOfRangeException: Could not extract an object Id from the Url 'https://drive.google.com/file/d/'. (Parameter 'uri') + at UiPath.GSuite.Models.CloudObjectIdentifier.FromUrl(Uri uri) + at UiPath.GSuite.Models.CloudObjectIdentifier.CreateFromUrlOrId(String urlOrId) + at UiPath.GSuite.Activities.GetFileInfo.ExecuteAsync(...) +``` + +## How this test reproduces it + +| Layer | Source | +|---|---| + +CLI-driven — no `process/` snapshot; the root cause is fully contained in the +`or jobs get` `Info` field. + +## Playbook validated + +`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/invalid-or-null-input.md` +(legacy `GetFileInfo` malformed-URL signature; also referenced from +`drive-file-not-found.md` and `overview.md`). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/2a5a382b83.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/2a5a382b83.json deleted file mode 100644 index 52593f6e69..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/2a5a382b83.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "StartTime": "2026-06-19T10:06:12.243Z", - "EndTime": "2026-06-19T10:06:18.743Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "1271f8d0-a443-46ce-b9fb-1fbb5da3105b", - "Info": "Could not extract an object Id from the Url 'https://drive.google.com/file/d/'. (Parameter 'uri')\r\n\r\nin Main.xaml\r\n at GetFileInfo \"Get File Info\"\r\n at Sequence \"Do\"\r\n at GSuiteApplicationScope \"Google Workspace Scope\"\r\n at Sequence \"Get File Info\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentOutOfRangeException: Could not extract an object Id from the Url 'https://drive.google.com/file/d/'. (Parameter 'uri')\r\n at UiPath.GSuite.Models.CloudObjectIdentifier.FromUrl(Uri uri)\r\n at UiPath.GSuite.Models.CloudObjectIdentifier.Initialize(String urlOrId)\r\n at UiPath.GSuite.Models.CloudObjectIdentifier..ctor(String urlOrId)\r\n at UiPath.GSuite.Models.CloudObjectIdentifier.CreateFromUrlOrId(String urlOrId)\r\n at UiPath.GSuite.Activities.GetFileInfo.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-19T10:06:10.397Z", - "ReleaseName": "DriveFileInspector", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53487, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-19T10:06:18.743Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "0e3938d5e2b0480b99d5bcaf0e9123d6", - "Id": 4015198 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/42e4c5d54e.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/42e4c5d54e.json deleted file mode 100644 index f187b3db0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/42e4c5d54e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-19T10:06:18.780Z", - "JobId": 4015198 - }, - { - "State": "Running", - "Timestamp": "2026-06-19T10:06:12.260Z", - "JobId": 4015198 - }, - { - "State": "Pending", - "Timestamp": "2026-06-19T10:06:10.443Z", - "JobId": 4015198 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/790c63a18f.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/790c63a18f.json deleted file mode 100644 index 17128079bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/790c63a18f.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "510ef231-2b59-4468-8808-141c9e6ce8bf", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-drive", - "ConnectorName": "Google Drive", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714201, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:14:22.110Z", - "Updated": "2026-06-10T11:48:03.921Z" - }, - { - "Id": "b03d73b7-611a-4e0e-b109-a85c10be40c8", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-sheets", - "ConnectorName": "Google Sheets", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714202, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:15:01.554Z", - "Updated": "2026-06-10T11:48:31.207Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/912f12e628.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/912f12e628.json deleted file mode 100644 index 19a96bd4a4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/912f12e628.json +++ /dev/null @@ -1,334 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "State": "Faulted", - "ProcessName": "DriveFileInspector", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-19T10:06:12.243Z", - "EndTime": "2026-06-19T10:06:18.743Z", - "CreationTime": "2026-06-19T10:06:10.397Z" - }, - { - "Key": "3ce81f18-b21e-4bd6-bbea-09af96b6353e", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-16T10:50:28.570Z", - "EndTime": "2026-06-16T10:51:03.023Z", - "CreationTime": "2026-06-16T10:50:28.570Z" - }, - { - "Key": "6f19905d-1832-4e4a-8b76-d18afc697183", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-16T10:35:23.557Z", - "EndTime": "2026-06-16T10:35:25.643Z", - "CreationTime": "2026-06-16T10:35:23.557Z" - }, - { - "Key": "6c8b1949-2603-437d-86a6-6b82b989fd6a", - "State": "Faulted", - "ProcessName": "DriveFileCreator", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T15:58:40.333Z", - "EndTime": "2026-06-12T15:58:44.587Z", - "CreationTime": "2026-06-12T15:58:39.873Z" - }, - { - "Key": "2fbf6ba4-b347-462f-a83e-33c6c34bbea6", - "State": "Faulted", - "ProcessName": "SheetTabBuilder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T15:34:04.717Z", - "EndTime": "2026-06-12T15:34:08.643Z", - "CreationTime": "2026-06-12T15:34:03.737Z" - }, - { - "Key": "d5e0e6b0-55b8-4256-8494-c601cb986dab", - "State": "Faulted", - "ProcessName": "SheetRangeWriter", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T15:11:29.767Z", - "EndTime": "2026-06-12T15:11:34.253Z", - "CreationTime": "2026-06-12T15:11:29.410Z" - }, - { - "Key": "4e00b4ca-b35b-4457-bb12-74f270d20cb0", - "State": "Faulted", - "ProcessName": "WorkspaceSheetSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T14:38:50.020Z", - "EndTime": "2026-06-12T14:38:56.633Z", - "CreationTime": "2026-06-12T14:38:49.027Z" - }, - { - "Key": "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "State": "Faulted", - "ProcessName": "SheetExportRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:59:41.830Z", - "EndTime": "2026-06-12T13:59:48.663Z", - "CreationTime": "2026-06-12T13:59:41.113Z" - }, - { - "Key": "f41ed418-163b-487b-8077-e51f8af4a1af", - "State": "Faulted", - "ProcessName": "DriveDocSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:21:01.380Z", - "EndTime": "2026-06-12T13:21:12.450Z", - "CreationTime": "2026-06-12T13:21:00.577Z" - }, - { - "Key": "5242daf1-a4e9-41d2-b828-3b70e9d59cc1", - "State": "Faulted", - "ProcessName": "InboxSearchBot", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T09:49:32.447Z", - "EndTime": "2026-06-11T09:49:41.220Z", - "CreationTime": "2026-06-11T09:49:31.683Z" - }, - { - "Key": "7fd5e890-f162-4591-a895-e649b09aea02", - "State": "Faulted", - "ProcessName": "DocumentUploader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:15:47.690Z", - "EndTime": "2026-06-11T08:15:54.983Z", - "CreationTime": "2026-06-11T08:15:46.967Z" - }, - { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "State": "Faulted", - "ProcessName": "TeamMailRetriever", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "CreationTime": "2026-06-11T08:01:50.607Z" - }, - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - } - ], - "Pagination": { - "Returned": 23, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/README.md deleted file mode 100644 index 6297ec31fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# GSuite Get File Info Malformed URL Troubleshooting — Faithful Replay - -This scenario replays a real `uipath-troubleshoot` investigation of a Faulted -Orchestrator job whose legacy UiPath.GSuite **Get File Info** activity threw a -**client-side** malformed-URL validation error — before any Google Drive API -call. - -## What the original session uncovered - -The user asked only *"why did my last job from folder Shared fail?"*. The agent: - -1. Listed folders → resolved **Shared** to key `1965a46b-db4e-469e-aaaa-7e0b379cb34d`. -2. Listed Faulted jobs in Shared → newest is **DriveFileInspector**, - job `0e3938d5-e2b0-480b-99d5-bcaf0e9123d6`. -3. Read the job's `Info` → a `System.ArgumentOutOfRangeException` thrown from - `CloudObjectIdentifier.FromUrl` / `CreateFromUrlOrId` inside - `UiPath.GSuite.Activities.GetFileInfo.ExecuteAsync`. - -Conclusion: a **client-side input-validation** failure — the File ID input was -a malformed Drive URL (`https://drive.google.com/file/d/`, truncated, no -`` segment), so the activity threw *before* any request reached Google -Drive (no HTTP status, no API round-trip). *Not* a Google API, 404, connection, -auth, scope, throttling, outage, or transient fault. (Listing connections -confirms both GSuite connections serving Shared are Enabled and valid — a -defensive fixture reinforces that the connection is fine.) - -## Client-side malformed-URL signature - -``` -System.ArgumentOutOfRangeException: Could not extract an object Id from the Url 'https://drive.google.com/file/d/'. (Parameter 'uri') - at UiPath.GSuite.Models.CloudObjectIdentifier.FromUrl(Uri uri) - at UiPath.GSuite.Models.CloudObjectIdentifier.CreateFromUrlOrId(String urlOrId) - at UiPath.GSuite.Activities.GetFileInfo.ExecuteAsync(...) -``` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven dispatcher) | -| `fixtures/mocks/responses/*.json` | verbatim, scrubbed stdout from the real session's `.local/investigations/raw/` | -| `fixtures/mocks/responses/manifest.json` | first-match-wins dispatch table; both trace forms covered; `docsai ask` is passthrough | - -CLI-driven — no `process/` snapshot; the root cause is fully contained in the -`or jobs get` `Info` field. - -## Playbook validated - -`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/invalid-or-null-input.md` -(legacy `GetFileInfo` malformed-URL signature; also referenced from -`drive-file-not-found.md` and `overview.md`). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/f6b9fc4e33.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/f6b9fc4e33.json deleted file mode 100644 index 33cab91370..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/m/r/f6b9fc4e33.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error reading job traces response", - "Instructions": "Unexpected token '<', \" MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com incl. the '...'s workspace' folder name, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving both GSuite connections serving Shared are Enabled/valid (the fault is NOT a connection problem). Both trace forms (`or jobs traces` and `traces spans get --job-key`) return the real non-JSON Failure envelope the live CLI produces (traces unavailable for this job); the client-side root cause is fully in `or jobs get` Info regardless.", - "rules": [ - { - "match": "or jobs get 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "file": "2a5a382b83.json", - "exit_code": 0 - }, - { - "match": "or jobs history 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "file": "42e4c5d54e.json", - "exit_code": 0 - }, - { - "match": "or jobs logs 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6 --level Error", - "file": "faa3f69e07.json", - "exit_code": 0 - }, - { - "match": "or jobs logs 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "file": "faa3f69e07.json", - "exit_code": 0 - }, - { - "match": "or jobs traces 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "traces spans get --job-key 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "912f12e628.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "912f12e628.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "912f12e628.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe. The client-side root cause is fully visible in `or jobs get` Info." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (0e3938d5 DriveFileInspector is the latest by StartTime)." - }, - { - "pattern": "or jobs get 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the client-side System.ArgumentOutOfRangeException from CloudObjectIdentifier.CreateFromUrlOrId in legacy GetFileInfo." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL — Agent may list connections to rule out a connection/auth problem; the fixture shows both GSuite connections Enabled. The fault is client-side input validation, not a connection issue. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces 0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", - "min": 0, - "description": "OPTIONAL — traces are unavailable for this job (real non-JSON Failure envelope); tracked-but-not-required. The synchronous client-side cause is fully in `or jobs get` Info." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/uip-fixture.json new file mode 100644 index 0000000000..9f7394b926 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-getfileinfo-malformed-url/data/uip-fixture.json @@ -0,0 +1,161 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"0e3938d5-e2b0-480b-99d5-bcaf0e9123d6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFileInspector\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-19T10:06:12.243Z\",\r\n \"EndTime\": \"2026-06-19T10:06:18.743Z\",\r\n \"CreationTime\": \"2026-06-19T10:06:10.397Z\"\r\n },\r\n {\r\n \"Key\": \"3ce81f18-b21e-4bd6-bbea-09af96b6353e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T10:50:28.570Z\",\r\n \"EndTime\": \"2026-06-16T10:51:03.023Z\",\r\n \"CreationTime\": \"2026-06-16T10:50:28.570Z\"\r\n },\r\n {\r\n \"Key\": \"6f19905d-1832-4e4a-8b76-d18afc697183\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T10:35:23.557Z\",\r\n \"EndTime\": \"2026-06-16T10:35:25.643Z\",\r\n \"CreationTime\": \"2026-06-16T10:35:23.557Z\"\r\n },\r\n {\r\n \"Key\": \"6c8b1949-2603-437d-86a6-6b82b989fd6a\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFileCreator\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:58:40.333Z\",\r\n \"EndTime\": \"2026-06-12T15:58:44.587Z\",\r\n \"CreationTime\": \"2026-06-12T15:58:39.873Z\"\r\n },\r\n {\r\n \"Key\": \"2fbf6ba4-b347-462f-a83e-33c6c34bbea6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetTabBuilder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:34:04.717Z\",\r\n \"EndTime\": \"2026-06-12T15:34:08.643Z\",\r\n \"CreationTime\": \"2026-06-12T15:34:03.737Z\"\r\n },\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 23,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "0e3938d5-e2b0-480b-99d5-bcaf0e9123d6", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T10:06:18.324Z\",\r\n \"Message\": \"Get File Info: Could not extract an object Id from the Url 'https://drive.google.com/file/d/'. (Parameter 'uri')\\r\\nin Main.xaml\\r\\n at GetFileInfo \\\"Get File Info\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at GSuiteApplicationScope \\\"Google Workspace Scope\\\"\\r\\n at Sequence \\\"Get File Info\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentOutOfRangeException\\r\\n at UiPath.GSuite.Models.CloudObjectIdentifier.FromUrl(Uri uri)\\r\\n at UiPath.GSuite.Models.CloudObjectIdentifier.Initialize(String urlOrId)\\r\\n at UiPath.GSuite.Models.CloudObjectIdentifier..ctor(String urlOrId)\\r\\n at UiPath.GSuite.Models.CloudObjectIdentifier.CreateFromUrlOrId(String urlOrId)\\r\\n at UiPath.GSuite.Activities.GetFileInfo.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "0e3938d5-e2b0-480b-99d5-bcaf0e9123d6" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\" UiPath.GSuite.Exceptions.GSuiteInternalException: is not a valid Spreadsheet range. (Parameter 'range') + ---> System.ArgumentNullException: is not a valid Spreadsheet range. (Parameter 'range') +``` + +## How this test reproduces it + +| Layer | Source | +|---|---| + +CLI-driven — no `process/` snapshot; the root cause is fully contained in the +`or jobs get` `Info` field. + +## Playbook validated + +`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/invalid-or-null-input.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/4af449581b.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/4af449581b.json deleted file mode 100644 index d00746906b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/4af449581b.json +++ /dev/null @@ -1,390 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "State": "Faulted", - "ProcessName": "SheetExportRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:59:41.830Z", - "EndTime": "2026-06-12T13:59:48.663Z", - "CreationTime": "2026-06-12T13:59:41.113Z" - }, - { - "Key": "f41ed418-163b-487b-8077-e51f8af4a1af", - "State": "Faulted", - "ProcessName": "DriveDocSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-12T13:21:01.380Z", - "EndTime": "2026-06-12T13:21:12.450Z", - "CreationTime": "2026-06-12T13:21:00.577Z" - }, - { - "Key": "5242daf1-a4e9-41d2-b828-3b70e9d59cc1", - "State": "Faulted", - "ProcessName": "InboxSearchBot", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T09:49:32.447Z", - "EndTime": "2026-06-11T09:49:41.220Z", - "CreationTime": "2026-06-11T09:49:31.683Z" - }, - { - "Key": "7fd5e890-f162-4591-a895-e649b09aea02", - "State": "Faulted", - "ProcessName": "DocumentUploader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:15:47.690Z", - "EndTime": "2026-06-11T08:15:54.983Z", - "CreationTime": "2026-06-11T08:15:46.967Z" - }, - { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "State": "Faulted", - "ProcessName": "TeamMailRetriever", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "CreationTime": "2026-06-11T08:01:50.607Z" - }, - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 27, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/57d27896d5.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/57d27896d5.json deleted file mode 100644 index 76909d4546..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/57d27896d5.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-12T13:59:48.723Z", - "JobId": 4015139 - }, - { - "State": "Running", - "Timestamp": "2026-06-12T13:59:41.870Z", - "JobId": 4015139 - }, - { - "State": "Pending", - "Timestamp": "2026-06-12T13:59:41.153Z", - "JobId": 4015139 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/775574d327.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/775574d327.json deleted file mode 100644 index f48ad4bd50..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/775574d327.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "StartTime": "2026-06-12T13:59:41.830Z", - "EndTime": "2026-06-12T13:59:48.663Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "18f51054-c262-48e9-a08a-d4b2ed2dbbc6", - "Info": "Value cannot be null. (Parameter 'range')\r\n\r\nin Main.xaml\r\n at WriteRangeConnections \"Write Range\"\r\n at Sequence \"Write Range with empty range repro\"\r\n at Main \"Main\"\r\n\r\nUiPath.GSuite.Exceptions.GSuiteException: Value cannot be null. (Parameter 'range')\r\n ---> UiPath.GSuite.Exceptions.GSuiteInternalException: is not a valid Spreadsheet range. (Parameter 'range')\r\n ---> System.ArgumentNullException: is not a valid Spreadsheet range. (Parameter 'range')\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-12T13:59:41.113Z", - "ReleaseName": "SheetExportRunner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53479, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T13:59:48.663Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c123367e8e5e4ec09903fd75d5ef9ad7", - "Id": 4015139 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/790c63a18f.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/790c63a18f.json deleted file mode 100644 index 17128079bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/790c63a18f.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "510ef231-2b59-4468-8808-141c9e6ce8bf", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-drive", - "ConnectorName": "Google Drive", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714201, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:14:22.110Z", - "Updated": "2026-06-10T11:48:03.921Z" - }, - { - "Id": "b03d73b7-611a-4e0e-b109-a85c10be40c8", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-google-sheets", - "ConnectorName": "Google Sheets", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "Yes", - "ByoaConnection": "No", - "ElementInstanceId": 7714202, - "Folder": "Shared", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Created": "2026-04-02T09:15:01.554Z", - "Updated": "2026-06-10T11:48:31.207Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/README.md deleted file mode 100644 index f589d2c51f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# GSuite Invalid / Null Input Troubleshooting — Faithful Replay - -This scenario replays a real `uipath-troubleshoot` investigation of a Faulted -Orchestrator job whose UiPath.GSuite **Write Range** activity threw a -**client-side** null/empty `range` validation error — before any Google -Sheets API call. - -## What the original session uncovered - -The user asked only *"why did my last job from folder Shared fail?"*. The agent: - -1. Listed folders → resolved **Shared** to key `1965a46b-db4e-469e-aaaa-7e0b379cb34d`. -2. Listed Faulted jobs in Shared → newest is **SheetExportRunner**, - job `c123367e-8e5e-4ec0-9903-fd75d5ef9ad7`. -3. Read the job's `Info` → a `UiPath.GSuite.Exceptions.GSuiteException` - wrapping `GSuiteInternalException` / `System.ArgumentNullException` on the - `range` parameter, thrown from `WriteRangeConnections`. - -Conclusion: a **client-side input-validation** failure — the `range` input was -null/empty, so the activity threw `Value cannot be null. (Parameter 'range')` -*before* any request reached Google Sheets (no HTTP status, no API round-trip). -*Not* a Google API, 404, connection, auth, scope, throttling, outage, or -transient fault. (Listing connections confirms both GSuite connections serving -Shared are Enabled and valid — a defensive fixture reinforces that the -connection is fine.) - -## Client-side null-range signature - -``` -UiPath.GSuite.Exceptions.GSuiteException: Value cannot be null. (Parameter 'range') - ---> UiPath.GSuite.Exceptions.GSuiteInternalException: is not a valid Spreadsheet range. (Parameter 'range') - ---> System.ArgumentNullException: is not a valid Spreadsheet range. (Parameter 'range') -``` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven dispatcher) | -| `fixtures/mocks/responses/*.json` | verbatim, scrubbed stdout from the real session's `.local/investigations/raw/` | -| `fixtures/mocks/responses/manifest.json` | first-match-wins dispatch table; `docsai ask` is passthrough | - -CLI-driven — no `process/` snapshot; the root cause is fully contained in the -`or jobs get` `Info` field. - -## Playbook validated - -`skills/uipath-troubleshoot/references/activity-packages/gsuite-activities/playbooks/invalid-or-null-input.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/aa4adc6d7a.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/aa4adc6d7a.json deleted file mode 100644 index 7d0f8a9644..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/aa4adc6d7a.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-12T13:59:48.285Z", - "Message": "Write Range: Value cannot be null. (Parameter 'range')\r\nin Main.xaml\r\n at WriteRangeConnections \"Write Range\"\r\n at Sequence \"Write Range with empty range repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.GSuite.Exceptions.GSuiteException\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/f6b9fc4e33.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/f6b9fc4e33.json deleted file mode 100644 index 33cab91370..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/m/r/f6b9fc4e33.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error reading job traces response", - "Instructions": "Unexpected token '<', \" MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com incl. the '...'s workspace' folder name, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving the GSuite connections are valid/Enabled (the fault is NOT a connection problem). `or jobs traces` returns the real non-JSON Failure envelope the live CLI produces (traces unavailable for this job); the client-side root cause is fully in `or jobs get` Info regardless.", - "rules": [ - { - "match": "or jobs get c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "file": "775574d327.json", - "exit_code": 0 - }, - { - "match": "or jobs history c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "file": "57d27896d5.json", - "exit_code": 0 - }, - { - "match": "or jobs logs c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "file": "aa4adc6d7a.json", - "exit_code": 0 - }, - { - "match": "or jobs traces c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "4af449581b.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "4af449581b.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "4af449581b.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe (e.g. a traces lookup, for which no raw was captured)." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (c123367e SheetExportRunner is the latest by StartTime)." - }, - { - "pattern": "or jobs get c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the client-side GSuiteException / ArgumentNullException chain on the `range` parameter." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL — Agent may list connections to rule out a connection/auth problem; the fixture shows both GSuite connections Enabled. The fault is client-side input validation, not a connection issue. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces c123367e-8e5e-4ec0-9903-fd75d5ef9ad7", - "min": 0, - "description": "OPTIONAL — traces are unavailable for this job (real non-JSON Failure envelope); tracked-but-not-required. The client-side cause is fully in `or jobs get` Info." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/uip-fixture.json new file mode 100644 index 0000000000..5659134416 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-invalid-or-null-input/data/uip-fixture.json @@ -0,0 +1,136 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 27,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 27,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"18f51054-c262-48e9-a08a-d4b2ed2dbbc6\",\r\n \"Info\": \"Value cannot be null. (Parameter 'range')\\r\\n\\r\\nin Main.xaml\\r\\n at WriteRangeConnections \\\"Write Range\\\"\\r\\n at Sequence \\\"Write Range with empty range repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.GSuite.Exceptions.GSuiteException: Value cannot be null. (Parameter 'range')\\r\\n ---> UiPath.GSuite.Exceptions.GSuiteInternalException: is not a valid Spreadsheet range. (Parameter 'range')\\r\\n ---> System.ArgumentNullException: is not a valid Spreadsheet range. (Parameter 'range')\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\",\r\n \"ReleaseName\": \"SheetExportRunner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53479,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c123367e8e5e4ec09903fd75d5ef9ad7\",\r\n \"Id\": 4015139\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T13:59:48.723Z\",\r\n \"JobId\": 4015139\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T13:59:41.870Z\",\r\n \"JobId\": 4015139\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-12T13:59:41.153Z\",\r\n \"JobId\": 4015139\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T13:59:48.285Z\",\r\n \"Message\": \"Write Range: Value cannot be null. (Parameter 'range')\\r\\nin Main.xaml\\r\\n at WriteRangeConnections \\\"Write Range\\\"\\r\\n at Sequence \\\"Write Range with empty range repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.GSuite.Exceptions.GSuiteException\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "c123367e-8e5e-4ec0-9903-fd75d5ef9ad7" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\" UiPath.GSuite.Exceptions.GSuiteInternalException: The request is not as expected by Google API.\r\n ---> Google.GoogleApiException: The service sheets has thrown an exception. HttpStatusCode is BadRequest. Unable to parse range: Sheet1!A0\r\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 256\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 217\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.A1NotationToGridRangeAsync(SheetsService service, String spreadsheetId, String a1RangeNotation, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, ISpreadsheetFile spreadsheetFile, RangeType rangeType, CancellationToken ct)\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 256\r\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\Apiary\\2023-06-26.09-20-33\\Src\\Support\\Google.Apis\\Requests\\ClientServiceRequest.cs:line 217\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.A1NotationToGridRangeAsync(SheetsService service, String spreadsheetId, String a1RangeNotation, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, ISpreadsheetFile spreadsheetFile, RangeType rangeType, CancellationToken ct)\r\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-12T15:11:29.410Z", - "ReleaseName": "SheetRangeWriter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53481, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T15:11:34.253Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d5e0e6b055b842568494c601cb986dab", - "Id": 4015141 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-sheets-invalid-range/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-sheets-invalid-range/data/m/r/manifest.json deleted file mode 100644 index c6f68b945c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-sheets-invalid-range/data/m/r/manifest.json +++ /dev/null @@ -1,87 +0,0 @@ -{ - "version": 2, - "_doc": "Verbatim, scrubbed responses captured from the original uipath-troubleshoot session that diagnosed a Faulted GSuite Write Range job in folder Shared. Job/folder keys, the malformed A1 range (Sheet1!A0), and all exception/stack text are preserved exactly; hostname, email, and Windows account were scrubbed (UIP-PW06WJSK -> MOCK-HOST, dan.morosanu@uipath.com -> original_email@test.com, UIPATH\\DAN.MOROSANU -> UIPATH\\REPLACEMENT_USER). First-match-wins: specific patterns precede generic ones. `is connections list` is a DEFENSIVE fixture proving both Shared GSuite connections are valid/Enabled (the fault is NOT a connection problem). `or jobs traces` returns the real non-JSON Failure envelope the live CLI produced (traces unavailable).", - "rules": [ - { - "match": "or jobs get d5e0e6b0-55b8-4256-8494-c601cb986dab", - "file": "fcf246d14d.json", - "exit_code": 0 - }, - { - "match": "or jobs history d5e0e6b0-55b8-4256-8494-c601cb986dab", - "file": "782bacb1c9.json", - "exit_code": 0 - }, - { - "match": "or jobs logs d5e0e6b0-55b8-4256-8494-c601cb986dab", - "file": "f31bf1e26d.json", - "exit_code": 0 - }, - { - "match": "or jobs traces d5e0e6b0-55b8-4256-8494-c601cb986dab", - "file": "f6b9fc4e33.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "aac6832948.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "aac6832948.json", - "exit_code": 0 - }, - { - "match": "or jobs list", - "file": "aac6832948.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "897a8be769.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "790c63a18f.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]", - "exit_code": 0, - "_doc": "Any command with no matching rule returns an empty JSON array with exit 0, mirroring an empty list result. Keeps the agent from hard-failing on an unrecorded probe." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent lists folders to resolve folder Shared to its key (1965a46b-...)." - }, - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "min": 1, - "description": "Agent finds the most-recent Faulted job in Shared (d5e0e6b0 is the latest by StartTime)." - }, - { - "pattern": "or jobs get d5e0e6b0-55b8-4256-8494-c601cb986dab", - "min": 1, - "description": "Agent reads the faulted job's Info, exposing the GSuiteException / Google 400 BadRequest stack and the malformed range Sheet1!A0." - }, - { - "pattern": "is connections list", - "min": 0, - "description": "OPTIONAL — Agent may list connections to rule out a connection/auth problem; the fixture shows both GSuite connections Enabled. Tracked-but-not-required." - }, - { - "pattern": "or jobs traces d5e0e6b0-55b8-4256-8494-c601cb986dab", - "min": 0, - "description": "OPTIONAL — traces are unavailable for this job (non-JSON Failure envelope); tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-sheets-invalid-range/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-sheets-invalid-range/data/uip-fixture.json new file mode 100644 index 0000000000..54e694c65f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/gsuite-sheets-invalid-range/data/uip-fixture.json @@ -0,0 +1,136 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 29,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetRangeWriter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\"\r\n },\r\n {\r\n \"Key\": \"4e00b4ca-b35b-4457-bb12-74f270d20cb0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WorkspaceSheetSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T14:38:50.020Z\",\r\n \"EndTime\": \"2026-06-12T14:38:56.633Z\",\r\n \"CreationTime\": \"2026-06-12T14:38:49.027Z\"\r\n },\r\n {\r\n \"Key\": \"c123367e-8e5e-4ec0-9903-fd75d5ef9ad7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SheetExportRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:59:41.830Z\",\r\n \"EndTime\": \"2026-06-12T13:59:48.663Z\",\r\n \"CreationTime\": \"2026-06-12T13:59:41.113Z\"\r\n },\r\n {\r\n \"Key\": \"f41ed418-163b-487b-8077-e51f8af4a1af\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveDocSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T13:21:01.380Z\",\r\n \"EndTime\": \"2026-06-12T13:21:12.450Z\",\r\n \"CreationTime\": \"2026-06-12T13:21:00.577Z\"\r\n },\r\n {\r\n \"Key\": \"5242daf1-a4e9-41d2-b828-3b70e9d59cc1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxSearchBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T09:49:32.447Z\",\r\n \"EndTime\": \"2026-06-11T09:49:41.220Z\",\r\n \"CreationTime\": \"2026-06-11T09:49:31.683Z\"\r\n },\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 29,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d5e0e6b0-55b8-4256-8494-c601cb986dab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d5e0e6b0-55b8-4256-8494-c601cb986dab\",\r\n \"StartTime\": \"2026-06-12T15:11:29.767Z\",\r\n \"EndTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"3af97bde-5437-4538-8a84-343973936067\",\r\n \"Info\": \"The request is not as expected by Google API.\\r\\n\\r\\nin Main.xaml\\r\\n at WriteRangeConnections \\\"Write Range\\\"\\r\\n at Sequence \\\"Write Range\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.GSuite.Exceptions.GSuiteException: The request is not as expected by Google API.\\r\\n ---> UiPath.GSuite.Exceptions.GSuiteInternalException: The request is not as expected by Google API.\\r\\n ---> Google.GoogleApiException: The service sheets has thrown an exception. HttpStatusCode is BadRequest. Unable to parse range: Sheet1!A0\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 256\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 217\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.A1NotationToGridRangeAsync(SheetsService service, String spreadsheetId, String a1RangeNotation, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, ISpreadsheetFile spreadsheetFile, RangeType rangeType, CancellationToken ct)\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 256\\r\\n at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken) in C:\\\\Apiary\\\\2023-06-26.09-20-33\\\\Src\\\\Support\\\\Google.Apis\\\\Requests\\\\ClientServiceRequest.cs:line 217\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.A1NotationToGridRangeAsync(SheetsService service, String spreadsheetId, String a1RangeNotation, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, ISpreadsheetFile spreadsheetFile, RangeType rangeType, CancellationToken ct)\\r\\n at UiPath.GSuite.Sheets.Extensions.SheetsServiceExtensions.GetRangeMetadataInformationAsync(SheetsService service, String spreadsheetId, String range, RangeType rangeType, CancellationToken ct)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, Boolean useFirstRowAsHeaders, RangeMetadata rangeMetadata, String[] columnNames, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.FromRequest(BaseRangeRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.<>c__DisplayClass53_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.TelemetryService.NullTelemetryService.OperationContextAsync(Func`2 method)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeAsync(BaseWriteRangeRequest writeRequest, Boolean includeHeaders, Int32 rowPosition, Int32 columnPosition, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Sheets.Services.SheetsServiceProxy.WriteRangeV2Async(WriteRangeRequest writeRangeRequest, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.WriteRangeConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-12T15:11:29.410Z\",\r\n \"ReleaseName\": \"SheetRangeWriter\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53481,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T15:11:34.253Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d5e0e6b055b842568494c601cb986dab\",\r\n \"Id\": 4015141\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "d5e0e6b0-55b8-4256-8494-c601cb986dab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-12T15:11:34.260Z\",\r\n \"JobId\": 4015141\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-12T15:11:29.777Z\",\r\n \"JobId\": 4015141\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-12T15:11:29.427Z\",\r\n \"JobId\": 4015141\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d5e0e6b0-55b8-4256-8494-c601cb986dab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T15:11:33.834Z\",\r\n \"Message\": \"Write Range: The request is not as expected by Google API.\\r\\nin Main.xaml\\r\\n at WriteRangeConnections \\\"Write Range\\\"\\r\\n at Sequence \\\"Write Range\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.GSuite.Exceptions.GSuiteException\\r\\n at UiPath.GSuite.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.GSuite.Activities.GoogleBaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "d5e0e6b0-55b8-4256-8494-c601cb986dab" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error reading job traces response\",\r\n \"Instructions\": \"Unexpected token '<', \\\"d__24.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.<>c__DisplayClass6_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-29T14:22:09.000Z", - "ReleaseName": "OrderNotifier", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-22", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10011, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-29T14:22:12.140Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c1d2e3f490114abc9def000000000011", - "Id": 4016011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/78f94b6048.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/README.md deleted file mode 100644 index 0b7a6836d4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# Broadcast Message Timeout — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.TimeoutException` thrown by a **`Broadcast Message`** -activity (`UiPath.IPC.Activities`) in `Main.xaml`. The agent runs the -`uipath-troubleshoot` skill against a `uip` CLI mock and must reach the same -root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Broadcast Message` activity **Broadcast Order Ready** broadcasts on channel -`OrderReadyChannel` with `TimeoutMS = 3000`. No **Message Receiver Trigger** was -listening on that channel in parallel at the broadcast instant, so the channel -search expired and the activity threw -`System.TimeoutException: Timeout of 3000 ms has passed and no channel was found -to send the message to.`. The fix is to run the receiver in parallel on the exact -same channel before broadcasting, match the channel string, and raise the timeout. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../../../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/activity-packages/ipc-activities/playbooks/broadcast-message-timeout.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/e2a5346a88.json deleted file mode 100644 index 54fa1a4a28..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-29T14:22:12.090Z", - "Message": "Broadcast Order Ready: Timeout of 3000 ms has passed and no channel was found to send the message to.\r\nin Main.xaml\r\n at Broadcast Message \"Broadcast Order Ready\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.TimeoutException: Timeout of 3000 ms has passed and no channel was found to send the message to.\r\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/manifest.json deleted file mode 100644 index 1a35c4aa20..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "78f94b6048.json" - }, - { - "match": "or jobs get c1d2e3f4-9011-4abc-9def-000000000011 --output json", - "file": "464db476bb.json" - }, - { - "match": "or jobs logs c1d2e3f4-9011-4abc-9def-000000000011 --level Error --output json", - "file": "e2a5346a88.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get c1d2e3f4-9011-4abc-9def-000000000011", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs c1d2e3f4-9011-4abc-9def-000000000011", - "min": 0, - "description": "OPTIONAL - agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/uip-fixture.json new file mode 100644 index 0000000000..40eb0ee831 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "c1d2e3f4-9011-4abc-9def-000000000011", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-29T14:22:12.090Z\",\r\n \"Message\": \"Broadcast Order Ready: Timeout of 3000 ms has passed and no channel was found to send the message to.\\r\\nin Main.xaml\\r\\n at Broadcast Message \\\"Broadcast Order Ready\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.TimeoutException: Timeout of 3000 ms has passed and no channel was found to send the message to.\\r\\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c1d2e3f4-9011-4abc-9def-000000000011", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c1d2e3f4-9011-4abc-9def-000000000011\",\r\n \"StartTime\": \"2026-06-29T14:22:09.000Z\",\r\n \"EndTime\": \"2026-06-29T14:22:12.140Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c1d2e3f4-9011-4abc-9def-000000000011\",\r\n \"Info\": \"Broadcast Order Ready: Timeout of 3000 ms has passed and no channel was found to send the message to.\\r\\n\\r\\nin Main.xaml\\r\\n at Broadcast Message \\\"Broadcast Order Ready\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: Timeout of 3000 ms has passed and no channel was found to send the message to.\\r\\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.<>c__DisplayClass6_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-29T14:22:09.000Z\",\r\n \"ReleaseName\": \"OrderNotifier\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-22\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10011,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-29T14:22:12.140Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c1d2e3f490114abc9def000000000011\",\r\n \"Id\": 4016011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get c1d2e3f4-9011-4abc-9def-000000000011", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs c1d2e3f4-9011-4abc-9def-000000000011", + "min": 0, + "description": "OPTIONAL - agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/task.yaml index a0d8b76efc..c873321429 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-timeout/task.yaml @@ -20,22 +20,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/README.md new file mode 100644 index 0000000000..dd8f2ffb5b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/README.md @@ -0,0 +1,34 @@ +# Broadcast Message Unauthorized Access — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.UnauthorizedAccessException` thrown by a +**`Broadcast Message`** activity (`UiPath.IPC.Activities`) in `Main.xaml`. The +agent runs the `uipath-troubleshoot` skill against a `uip` CLI mock and must +reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Broadcast Message` activity **Broadcast Control Signal** broadcasts on +channel `ControlChannel`. Opening the channel's local named-pipe endpoint +(`\\.\pipe\UiPath.Ipc.ControlChannel`) was refused by the pipe ACL, throwing +`System.UnauthorizedAccessException: Access to the path ... is denied.`. IPC is +confined to the same robot, user, session, and machine; the sender and receiver +were separated by a session / user / elevation boundary. The fix is to co-locate +both processes on the same robot / user / session / elevation. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/activity-packages/ipc-activities/playbooks/broadcast-message-unauthorized-access.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/464db476bb.json deleted file mode 100644 index 55957c1a31..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/464db476bb.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c1d2e3f4-9012-4abc-9def-000000000012", - "StartTime": "2026-06-29T16:04:31.000Z", - "EndTime": "2026-06-29T16:04:31.560Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c1d2e3f4-9012-4abc-9def-000000000012", - "Info": "Broadcast Control Signal: Access to the path '\\\\.\\pipe\\UiPath.Ipc.ControlChannel' is denied.\r\n\r\nin Main.xaml\r\n at Broadcast Message \"Broadcast Control Signal\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.UnauthorizedAccessException: Access to the path '\\\\.\\pipe\\UiPath.Ipc.ControlChannel' is denied.\r\n at System.IO.Pipes.NamedPipeClientStream.TryConnect(Int32 timeout)\r\n at System.IO.Pipes.NamedPipeClientStream.Connect(Int32 timeout)\r\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-29T16:04:31.000Z", - "ReleaseName": "SessionBridge", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-22", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10012, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-29T16:04:31.560Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c1d2e3f490124abc9def000000000012", - "Id": 4016012 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/78f94b6048.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/README.md deleted file mode 100644 index 494c25a2e3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# Broadcast Message Unauthorized Access — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.UnauthorizedAccessException` thrown by a -**`Broadcast Message`** activity (`UiPath.IPC.Activities`) in `Main.xaml`. The -agent runs the `uipath-troubleshoot` skill against a `uip` CLI mock and must -reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Broadcast Message` activity **Broadcast Control Signal** broadcasts on -channel `ControlChannel`. Opening the channel's local named-pipe endpoint -(`\\.\pipe\UiPath.Ipc.ControlChannel`) was refused by the pipe ACL, throwing -`System.UnauthorizedAccessException: Access to the path ... is denied.`. IPC is -confined to the same robot, user, session, and machine; the sender and receiver -were separated by a session / user / elevation boundary. The fix is to co-locate -both processes on the same robot / user / session / elevation. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../../../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/activity-packages/ipc-activities/playbooks/broadcast-message-unauthorized-access.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/e2a5346a88.json deleted file mode 100644 index 92629fe773..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-29T16:04:31.520Z", - "Message": "Broadcast Control Signal: Access to the path '\\\\.\\pipe\\UiPath.Ipc.ControlChannel' is denied.\r\nin Main.xaml\r\n at Broadcast Message \"Broadcast Control Signal\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.UnauthorizedAccessException: Access to the path '\\\\.\\pipe\\UiPath.Ipc.ControlChannel' is denied.\r\n at System.IO.Pipes.NamedPipeClientStream.TryConnect(Int32 timeout)\r\n at System.IO.Pipes.NamedPipeClientStream.Connect(Int32 timeout)\r\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/manifest.json deleted file mode 100644 index 82e3b6415d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "78f94b6048.json" - }, - { - "match": "or jobs get c1d2e3f4-9012-4abc-9def-000000000012 --output json", - "file": "464db476bb.json" - }, - { - "match": "or jobs logs c1d2e3f4-9012-4abc-9def-000000000012 --level Error --output json", - "file": "e2a5346a88.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get c1d2e3f4-9012-4abc-9def-000000000012", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs c1d2e3f4-9012-4abc-9def-000000000012", - "min": 0, - "description": "OPTIONAL - agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/uip-fixture.json new file mode 100644 index 0000000000..662e61fae6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "c1d2e3f4-9012-4abc-9def-000000000012", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-29T16:04:31.520Z\",\r\n \"Message\": \"Broadcast Control Signal: Access to the path '\\\\\\\\.\\\\pipe\\\\UiPath.Ipc.ControlChannel' is denied.\\r\\nin Main.xaml\\r\\n at Broadcast Message \\\"Broadcast Control Signal\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.UnauthorizedAccessException: Access to the path '\\\\\\\\.\\\\pipe\\\\UiPath.Ipc.ControlChannel' is denied.\\r\\n at System.IO.Pipes.NamedPipeClientStream.TryConnect(Int32 timeout)\\r\\n at System.IO.Pipes.NamedPipeClientStream.Connect(Int32 timeout)\\r\\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c1d2e3f4-9012-4abc-9def-000000000012", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c1d2e3f4-9012-4abc-9def-000000000012\",\r\n \"StartTime\": \"2026-06-29T16:04:31.000Z\",\r\n \"EndTime\": \"2026-06-29T16:04:31.560Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c1d2e3f4-9012-4abc-9def-000000000012\",\r\n \"Info\": \"Broadcast Control Signal: Access to the path '\\\\\\\\.\\\\pipe\\\\UiPath.Ipc.ControlChannel' is denied.\\r\\n\\r\\nin Main.xaml\\r\\n at Broadcast Message \\\"Broadcast Control Signal\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.UnauthorizedAccessException: Access to the path '\\\\\\\\.\\\\pipe\\\\UiPath.Ipc.ControlChannel' is denied.\\r\\n at System.IO.Pipes.NamedPipeClientStream.TryConnect(Int32 timeout)\\r\\n at System.IO.Pipes.NamedPipeClientStream.Connect(Int32 timeout)\\r\\n at UiPath.IPC.Activities.BroadcastMessage.d__24.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.System.Activities.IAsyncCodeActivity.FinishExecution(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-29T16:04:31.000Z\",\r\n \"ReleaseName\": \"SessionBridge\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-22\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10012,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-29T16:04:31.560Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c1d2e3f490124abc9def000000000012\",\r\n \"Id\": 4016012\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get c1d2e3f4-9012-4abc-9def-000000000012", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs c1d2e3f4-9012-4abc-9def-000000000012", + "min": 0, + "description": "OPTIONAL - agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/task.yaml index 03b2879237..a9dc7eabe4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/ipc-broadcast-message-unauthorized/task.yaml @@ -21,22 +21,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/README.md new file mode 100644 index 0000000000..25697768fe --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/README.md @@ -0,0 +1,50 @@ +# Jira Activity Could Not Be Loaded - RestSharp Dependency Conflict + +This scenario reproduces a `Jira Scope` job that faults at **activity load** with +a `System.IO.FileLoadException` on **RestSharp** (manifest definition does not +match the assembly reference, `HRESULT 0x80131040`). The legacy +`UiPath.Jira.Activities` pack pins `RestSharp 106.x`; a second package in the +project (`UiPath.WebAPI.Activities`) pulls a newer RestSharp, so the assembly +loaded at runtime no longer matches the Jira pack's reference and the activity +cannot bind. + +## What this scenario uncovers + +**Root Cause:** A transitive RestSharp version conflict. `project.json` depends on +both `UiPath.Jira.Activities` `[1.9.4]` and `UiPath.WebAPI.Activities` `[1.20.2]`; +only one RestSharp loads, and it is not the `106.15.0.0` the Jira pack was built +against. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-activity-missing-or-not-loaded.md` + +The fault is an assembly **load** failure at the scope, before authentication or +any REST call — so the authentication and response/URL families are explicitly +ruled out. The user reports it began after adding a second package, the classic +shared-dependency-collision signature. The user is framed as **off-host**, so the +correct agent behavior is to tie the load error to the RestSharp conflict in +`project.json` and recommend resolving the pin or migrating to the Integration +Service Jira connector - not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Create Issue` and a `project.json` depending on both `UiPath.Jira.Activities` and `UiPath.WebAPI.Activities` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-activity-missing-or-not-loaded.md` +- Agent identified a transitive RestSharp version conflict between the Jira pack + and another package (not a credential, URL, or Jira-side problem) and + recommended resolving the version conflict or migrating to the Integration + Service Jira connector, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/16478a1b40.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/16478a1b40.json deleted file mode 100644 index 19b17e665d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/16478a1b40.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b03c3d4-e5f6-4071-9b03-000000000003", - "StartTime": "2026-06-15T11:30:01.050Z", - "EndTime": "2026-06-15T11:30:02.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c03d4e5-f6a7-4182-9c03-000000000003", - "Info": "Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileLoadException: Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies.", - "CreationTime": "2026-06-15T11:30:00.600Z", - "ReleaseName": "JiraTicketCreator", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013303, - "OrganizationUnitId": 7791618, - "FolderKey": "5a03a2b3-d4e5-4f60-8a03-000000000003", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T11:30:02.640Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b03c3d4-e5f6-4071-9b03-000000000003", - "Id": 813303203 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/46ac4679cc.json deleted file mode 100644 index d84b930939..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a03a2b3-d4e5-4f60-8a03-000000000003", - "Name": "Integrations", - "Path": "Integrations", - "Description": "Production folder for Jira integration processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/6b32b09ab2.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/6b32b09ab2.json deleted file mode 100644 index 8205991a82..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/6b32b09ab2.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T11:30:02.510Z", - "Message": "[Jira Scope] Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T11:30:02.590Z", - "Message": "System.IO.FileLoadException: Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/8a610cf336.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/8a610cf336.json deleted file mode 100644 index a7cc36296c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/8a610cf336.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b03c3d4-e5f6-4071-9b03-000000000003", - "State": "Faulted", - "ProcessName": "JiraTicketCreator", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T11:30:01.050Z", - "EndTime": "2026-06-15T11:30:02.640Z", - "CreationTime": "2026-06-15T11:30:00.600Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/README.md deleted file mode 100644 index 1e4ab09388..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# Jira Activity Could Not Be Loaded - RestSharp Dependency Conflict - -This scenario reproduces a `Jira Scope` job that faults at **activity load** with -a `System.IO.FileLoadException` on **RestSharp** (manifest definition does not -match the assembly reference, `HRESULT 0x80131040`). The legacy -`UiPath.Jira.Activities` pack pins `RestSharp 106.x`; a second package in the -project (`UiPath.WebAPI.Activities`) pulls a newer RestSharp, so the assembly -loaded at runtime no longer matches the Jira pack's reference and the activity -cannot bind. - -## What this scenario uncovers - -**Root Cause:** A transitive RestSharp version conflict. `project.json` depends on -both `UiPath.Jira.Activities` `[1.9.4]` and `UiPath.WebAPI.Activities` `[1.20.2]`; -only one RestSharp loads, and it is not the `106.15.0.0` the Jira pack was built -against. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-activity-missing-or-not-loaded.md` - -The fault is an assembly **load** failure at the scope, before authentication or -any REST call — so the authentication and response/URL families are explicitly -ruled out. The user reports it began after adding a second package, the classic -shared-dependency-collision signature. The user is framed as **off-host**, so the -correct agent behavior is to tie the load error to the RestSharp conflict in -`project.json` and recommend resolving the pin or migrating to the Integration -Service Jira connector - not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Create Issue` and a `project.json` depending on both `UiPath.Jira.Activities` and `UiPath.WebAPI.Activities` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-activity-missing-or-not-loaded.md` -- Agent identified a transitive RestSharp version conflict between the Jira pack - and another package (not a credential, URL, or Jira-side problem) and - recommended resolving the version conflict or migrating to the Integration - Service Jira connector, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/bb52bff0ef.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/bb52bff0ef.json deleted file mode 100644 index 5a0c7a0e29..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/bb52bff0ef.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-15T11:30:01.220Z", - "Message": "JiraTicketCreator execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T11:30:02.510Z", - "Message": "[Jira Scope] Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T11:30:02.590Z", - "Message": "System.IO.FileLoadException: Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-15T11:30:02.640Z", - "Message": "JiraTicketCreator execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/manifest.json deleted file mode 100644 index a5868e07c0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a03a2b3-d4e5-4f60-8a03-000000000003\" --state Faulted --output json", - "file": "8a610cf336.json" - }, - { - "match": "or jobs list --folder-key 5a03a2b3-d4e5-4f60-8a03-000000000003 --state Faulted --output json", - "file": "8a610cf336.json" - }, - { - "match": "or jobs get \"5b03c3d4-e5f6-4071-9b03-000000000003\" --output json", - "file": "16478a1b40.json" - }, - { - "match": "or jobs get 5b03c3d4-e5f6-4071-9b03-000000000003 --output json", - "file": "16478a1b40.json" - }, - { - "match": "or jobs logs \"5b03c3d4-e5f6-4071-9b03-000000000003\" --level Error --output json", - "file": "6b32b09ab2.json" - }, - { - "match": "or jobs logs 5b03c3d4-e5f6-4071-9b03-000000000003 --level Error --output json", - "file": "6b32b09ab2.json" - }, - { - "match": "or jobs logs \"5b03c3d4-e5f6-4071-9b03-000000000003\" --output json", - "file": "bb52bff0ef.json" - }, - { - "match": "or jobs logs 5b03c3d4-e5f6-4071-9b03-000000000003 --output json", - "file": "bb52bff0ef.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/uip-fixture.json new file mode 100644 index 0000000000..86aa7175f3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a03a2b3-d4e5-4f60-8a03-000000000003", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b03c3d4-e5f6-4071-9b03-000000000003\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraTicketCreator\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T11:30:01.050Z\",\r\n \"EndTime\": \"2026-06-15T11:30:02.640Z\",\r\n \"CreationTime\": \"2026-06-15T11:30:00.600Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b03c3d4-e5f6-4071-9b03-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T11:30:02.510Z\",\r\n \"Message\": \"[Jira Scope] Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T11:30:02.590Z\",\r\n \"Message\": \"System.IO.FileLoadException: Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b03c3d4-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b03c3d4-e5f6-4071-9b03-000000000003\",\r\n \"StartTime\": \"2026-06-15T11:30:01.050Z\",\r\n \"EndTime\": \"2026-06-15T11:30:02.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c03d4e5-f6a7-4182-9c03-000000000003\",\r\n \"Info\": \"Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileLoadException: Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies.\",\r\n \"CreationTime\": \"2026-06-15T11:30:00.600Z\",\r\n \"ReleaseName\": \"JiraTicketCreator\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013303,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a03a2b3-d4e5-4f60-8a03-000000000003\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T11:30:02.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b03c3d4-e5f6-4071-9b03-000000000003\",\r\n \"Id\": 813303203\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b03c3d4-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-15T11:30:01.220Z\",\r\n \"Message\": \"JiraTicketCreator execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T11:30:02.510Z\",\r\n \"Message\": \"[Jira Scope] Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T11:30:02.590Z\",\r\n \"Message\": \"System.IO.FileLoadException: Could not load file or assembly 'RestSharp, Version=106.15.0.0, Culture=neutral, PublicKeyToken=598062e77f915f75' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-15T11:30:02.640Z\",\r\n \"Message\": \"JiraTicketCreator execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a03a2b3-d4e5-4f60-8a03-000000000003\",\r\n \"Name\": \"Integrations\",\r\n \"Path\": \"Integrations\",\r\n \"Description\": \"Production folder for Jira integration processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/task.yaml index 9723ff013b..a209f55e3d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-activity-restsharp-load/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/README.md new file mode 100644 index 0000000000..56930543e5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/README.md @@ -0,0 +1,48 @@ +# Jira Scope Authentication Failure - Username Is an accountId, Not the Email (A2) + +This scenario reproduces a `Jira Scope` authentication failure caused by the +`Username` being set to an alphanumeric Jira Cloud **accountId** instead of the +account **email**. Jira Cloud authenticates API-token calls against the account +email; an `accountId` there is rejected at scope open with `Authentication +information is invalid. Please check your credentials and try again.` + +## What this scenario uncovers + +**Root Cause:** The `Jira Scope` `Username` is `557058:9d8c7b6a-1f2e-4a3b-bc01-aabbccddeeff` +(a Jira `accountId`) rather than the account email. Every other auth property is +valid, so Atlassian rejects only the username identity. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-scope-authentication-failures.md` +sub-case **A2** (username must be the account email). + +The project isolates A2 by ruling out the sibling sub-cases: the `Api Token` is a +`SecureString` in-argument (not a plain `String`, so **not A1**), there is no +`Client Id` / `Client Secret` (**not A3**), and `Authentication Type = Api Token` +rather than basic password auth (**not A4**). The user is framed as **off-host**, +so the correct agent behavior is to tie the auth failure to the username format +and recommend the email - handing the user the Atlassian-side confirmation, not +attempting host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` (Api Token auth, SecureString token, accountId username) wrapping `Search Issues` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-scope-authentication-failures.md` (sub-case A2) +- Agent identified the `Username` being a Jira `accountId` instead of the account + email as the cause (not a bad token, leftover OAuth params, or an MFA blocker) + and recommended setting `Username` to the account email, without fabricating + host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/0959a65109.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/0959a65109.json deleted file mode 100644 index d7f5df57de..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/0959a65109.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b01c3d4-e5f6-4071-9b01-000000000001", - "StartTime": "2026-06-15T09:12:03.110Z", - "EndTime": "2026-06-15T09:12:04.905Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c01d4e5-f6a7-4182-9c01-000000000001", - "Info": "Jira Scope: Authentication information is invalid. Please check your credentials and try again.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: Authentication information is invalid. Please check your credentials and try again.", - "CreationTime": "2026-06-15T09:12:02.700Z", - "ReleaseName": "JiraIssueSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013301, - "OrganizationUnitId": 7791618, - "FolderKey": "5a01a2b3-d4e5-4f60-8a01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T09:12:04.905Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b01c3d4-e5f6-4071-9b01-000000000001", - "Id": 813301201 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/46ac4679cc.json deleted file mode 100644 index 5b239f1ff8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a01a2b3-d4e5-4f60-8a01-000000000001", - "Name": "Issue Sync", - "Path": "Issue Sync", - "Description": "Production folder for Jira issue-sync processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/48d996c6b7.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/48d996c6b7.json deleted file mode 100644 index 6427de3fca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/48d996c6b7.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T09:12:04.780Z", - "Message": "[Jira Scope] Authentication information is invalid. Please check your credentials and try again." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T09:12:04.860Z", - "Message": "UiPath.Jira.Activities.JiraException: Authentication information is invalid. Please check your credentials and try again.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/9e5b2139bc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/9e5b2139bc.json deleted file mode 100644 index 71667592a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/9e5b2139bc.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-15T09:12:03.260Z", - "Message": "JiraIssueSync execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T09:12:03.940Z", - "Message": "Jira Scope: opening session to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T09:12:04.780Z", - "Message": "[Jira Scope] Authentication information is invalid. Please check your credentials and try again." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T09:12:04.860Z", - "Message": "UiPath.Jira.Activities.JiraException: Authentication information is invalid. Please check your credentials and try again.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-15T09:12:04.905Z", - "Message": "JiraIssueSync execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/README.md deleted file mode 100644 index 9d9c7a2e26..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Jira Scope Authentication Failure - Username Is an accountId, Not the Email (A2) - -This scenario reproduces a `Jira Scope` authentication failure caused by the -`Username` being set to an alphanumeric Jira Cloud **accountId** instead of the -account **email**. Jira Cloud authenticates API-token calls against the account -email; an `accountId` there is rejected at scope open with `Authentication -information is invalid. Please check your credentials and try again.` - -## What this scenario uncovers - -**Root Cause:** The `Jira Scope` `Username` is `557058:9d8c7b6a-1f2e-4a3b-bc01-aabbccddeeff` -(a Jira `accountId`) rather than the account email. Every other auth property is -valid, so Atlassian rejects only the username identity. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-scope-authentication-failures.md` -sub-case **A2** (username must be the account email). - -The project isolates A2 by ruling out the sibling sub-cases: the `Api Token` is a -`SecureString` in-argument (not a plain `String`, so **not A1**), there is no -`Client Id` / `Client Secret` (**not A3**), and `Authentication Type = Api Token` -rather than basic password auth (**not A4**). The user is framed as **off-host**, -so the correct agent behavior is to tie the auth failure to the username format -and recommend the email - handing the user the Atlassian-side confirmation, not -attempting host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` (Api Token auth, SecureString token, accountId username) wrapping `Search Issues` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-scope-authentication-failures.md` (sub-case A2) -- Agent identified the `Username` being a Jira `accountId` instead of the account - email as the cause (not a bad token, leftover OAuth params, or an MFA blocker) - and recommended setting `Username` to the account email, without fabricating - host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/fa4348bd4b.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/fa4348bd4b.json deleted file mode 100644 index da7b7c68e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/fa4348bd4b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b01c3d4-e5f6-4071-9b01-000000000001", - "State": "Faulted", - "ProcessName": "JiraIssueSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T09:12:03.110Z", - "EndTime": "2026-06-15T09:12:04.905Z", - "CreationTime": "2026-06-15T09:12:02.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/manifest.json deleted file mode 100644 index 2b842420f9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a01a2b3-d4e5-4f60-8a01-000000000001\" --state Faulted --output json", - "file": "fa4348bd4b.json" - }, - { - "match": "or jobs list --folder-key 5a01a2b3-d4e5-4f60-8a01-000000000001 --state Faulted --output json", - "file": "fa4348bd4b.json" - }, - { - "match": "or jobs get \"5b01c3d4-e5f6-4071-9b01-000000000001\" --output json", - "file": "0959a65109.json" - }, - { - "match": "or jobs get 5b01c3d4-e5f6-4071-9b01-000000000001 --output json", - "file": "0959a65109.json" - }, - { - "match": "or jobs logs \"5b01c3d4-e5f6-4071-9b01-000000000001\" --level Error --output json", - "file": "48d996c6b7.json" - }, - { - "match": "or jobs logs 5b01c3d4-e5f6-4071-9b01-000000000001 --level Error --output json", - "file": "48d996c6b7.json" - }, - { - "match": "or jobs logs \"5b01c3d4-e5f6-4071-9b01-000000000001\" --output json", - "file": "9e5b2139bc.json" - }, - { - "match": "or jobs logs 5b01c3d4-e5f6-4071-9b01-000000000001 --output json", - "file": "9e5b2139bc.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/uip-fixture.json new file mode 100644 index 0000000000..cdc3145d6d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a01a2b3-d4e5-4f60-8a01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b01c3d4-e5f6-4071-9b01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraIssueSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T09:12:03.110Z\",\r\n \"EndTime\": \"2026-06-15T09:12:04.905Z\",\r\n \"CreationTime\": \"2026-06-15T09:12:02.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b01c3d4-e5f6-4071-9b01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T09:12:04.780Z\",\r\n \"Message\": \"[Jira Scope] Authentication information is invalid. Please check your credentials and try again.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T09:12:04.860Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Authentication information is invalid. Please check your credentials and try again.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b01c3d4-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b01c3d4-e5f6-4071-9b01-000000000001\",\r\n \"StartTime\": \"2026-06-15T09:12:03.110Z\",\r\n \"EndTime\": \"2026-06-15T09:12:04.905Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c01d4e5-f6a7-4182-9c01-000000000001\",\r\n \"Info\": \"Jira Scope: Authentication information is invalid. Please check your credentials and try again.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: Authentication information is invalid. Please check your credentials and try again.\",\r\n \"CreationTime\": \"2026-06-15T09:12:02.700Z\",\r\n \"ReleaseName\": \"JiraIssueSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013301,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a01a2b3-d4e5-4f60-8a01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T09:12:04.905Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b01c3d4-e5f6-4071-9b01-000000000001\",\r\n \"Id\": 813301201\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b01c3d4-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-15T09:12:03.260Z\",\r\n \"Message\": \"JiraIssueSync execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T09:12:03.940Z\",\r\n \"Message\": \"Jira Scope: opening session to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T09:12:04.780Z\",\r\n \"Message\": \"[Jira Scope] Authentication information is invalid. Please check your credentials and try again.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T09:12:04.860Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Authentication information is invalid. Please check your credentials and try again.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-15T09:12:04.905Z\",\r\n \"Message\": \"JiraIssueSync execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a01a2b3-d4e5-4f60-8a01-000000000001\",\r\n \"Name\": \"Issue Sync\",\r\n \"Path\": \"Issue Sync\",\r\n \"Description\": \"Production folder for Jira issue-sync processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/task.yaml index 5c7126367a..7c0b932489 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-auth-username-not-email/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/README.md new file mode 100644 index 0000000000..14f948b3de --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/README.md @@ -0,0 +1,49 @@ +# Add Comment Fails - 403 Forbidden / Missing Add Comments Permission (C3) + +This scenario reproduces an `Add Comment` failure caused by the bot account +lacking the **Add Comments** permission in the project. The Jira Scope +authenticates and the issue is located, but Jira rejects the comment with `403 +(Forbidden): You do not have permission to comment on this issue` - the account +is authenticated but not authorized for this action. + +## What this scenario uncovers + +**Root Cause:** `rpa-bot@company.com` authenticated successfully (valid API +token) but is not in a group / project role mapped to **Add Comments** in the +target project's permission scheme, so the comment is rejected with `403`. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-add-comment-failures.md` +sub-case **C3** (403 Forbidden / missing project permission). + +The key discriminator is `403` (authenticated, not authorized) vs `401` / +`Authentication information is invalid` (an auth failure). The `IssueKey` +(`OPS-1502`) is well-formed and reading works, so this is **not** the C2 +bad-key/`404` case; no `Visibility` is set, so it is **not** C4. The user is +framed as **off-host** and is not a Jira admin, so the correct agent behavior is +to identify the missing project permission and hand the user a precise admin +request - not to attempt host commands or workflow edits. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Add Comment` on a well-formed `IssueKey` (no Visibility set) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-add-comment-failures.md` (sub-case C3) +- Agent identified the missing **Add Comments** project permission for the + authenticated bot account as the cause (not an invalid credential, a bad + IssueKey, or a visibility issue) and recommended having a Jira administrator + grant the permission (verifiable by commenting manually as the service + account), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/31042d2c8d.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/31042d2c8d.json deleted file mode 100644 index 90e39a4baa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/31042d2c8d.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T14:02:03.810Z", - "Message": "[Add Comment] Response status code does not indicate success: 403 (Forbidden). You do not have permission to comment on this issue." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T14:02:03.880Z", - "Message": "UiPath.Jira.Activities.JiraException: You do not have permission to comment on this issue.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/46ac4679cc.json deleted file mode 100644 index 1730b7e2de..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a08a2b3-d4e5-4f60-8a08-000000000008", - "Name": "Issue Sync", - "Path": "Issue Sync", - "Description": "Production folder for Jira commenting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/88bb60105a.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/88bb60105a.json deleted file mode 100644 index 1262af90b5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/88bb60105a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b08c3d4-e5f6-4071-9b08-000000000008", - "StartTime": "2026-06-17T14:02:02.190Z", - "EndTime": "2026-06-17T14:02:03.940Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c08d4e5-f6a7-4182-9c08-000000000008", - "Info": "Add Comment: Response status code does not indicate success: 403 (Forbidden). You do not have permission to comment on this issue.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: You do not have permission to comment on this issue.", - "CreationTime": "2026-06-17T14:02:01.760Z", - "ReleaseName": "JiraCommentSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013308, - "OrganizationUnitId": 7791618, - "FolderKey": "5a08a2b3-d4e5-4f60-8a08-000000000008", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T14:02:03.940Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b08c3d4-e5f6-4071-9b08-000000000008", - "Id": 813308208 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/98d07e5c07.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/98d07e5c07.json deleted file mode 100644 index 9b0422a39b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/98d07e5c07.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b08c3d4-e5f6-4071-9b08-000000000008", - "State": "Faulted", - "ProcessName": "JiraCommentSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-17T14:02:02.190Z", - "EndTime": "2026-06-17T14:02:03.940Z", - "CreationTime": "2026-06-17T14:02:01.760Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/README.md deleted file mode 100644 index fc9c79277b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Add Comment Fails - 403 Forbidden / Missing Add Comments Permission (C3) - -This scenario reproduces an `Add Comment` failure caused by the bot account -lacking the **Add Comments** permission in the project. The Jira Scope -authenticates and the issue is located, but Jira rejects the comment with `403 -(Forbidden): You do not have permission to comment on this issue` - the account -is authenticated but not authorized for this action. - -## What this scenario uncovers - -**Root Cause:** `rpa-bot@company.com` authenticated successfully (valid API -token) but is not in a group / project role mapped to **Add Comments** in the -target project's permission scheme, so the comment is rejected with `403`. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-add-comment-failures.md` -sub-case **C3** (403 Forbidden / missing project permission). - -The key discriminator is `403` (authenticated, not authorized) vs `401` / -`Authentication information is invalid` (an auth failure). The `IssueKey` -(`OPS-1502`) is well-formed and reading works, so this is **not** the C2 -bad-key/`404` case; no `Visibility` is set, so it is **not** C4. The user is -framed as **off-host** and is not a Jira admin, so the correct agent behavior is -to identify the missing project permission and hand the user a precise admin -request - not to attempt host commands or workflow edits. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Add Comment` on a well-formed `IssueKey` (no Visibility set) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-add-comment-failures.md` (sub-case C3) -- Agent identified the missing **Add Comments** project permission for the - authenticated bot account as the cause (not an invalid credential, a bad - IssueKey, or a visibility issue) and recommended having a Jira administrator - grant the permission (verifiable by commenting manually as the service - account), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/e6645b8fc4.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/e6645b8fc4.json deleted file mode 100644 index d4ed475494..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/e6645b8fc4.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-17T14:02:02.380Z", - "Message": "JiraCommentSync execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T14:02:03.090Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T14:02:03.560Z", - "Message": "Add Comment: posting comment to issue OPS-1502 as rpa-bot@company.com" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T14:02:03.810Z", - "Message": "[Add Comment] Response status code does not indicate success: 403 (Forbidden). You do not have permission to comment on this issue." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T14:02:03.880Z", - "Message": "UiPath.Jira.Activities.JiraException: You do not have permission to comment on this issue.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-17T14:02:03.940Z", - "Message": "JiraCommentSync execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/manifest.json deleted file mode 100644 index c0018e9028..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a08a2b3-d4e5-4f60-8a08-000000000008\" --state Faulted --output json", - "file": "98d07e5c07.json" - }, - { - "match": "or jobs list --folder-key 5a08a2b3-d4e5-4f60-8a08-000000000008 --state Faulted --output json", - "file": "98d07e5c07.json" - }, - { - "match": "or jobs get \"5b08c3d4-e5f6-4071-9b08-000000000008\" --output json", - "file": "88bb60105a.json" - }, - { - "match": "or jobs get 5b08c3d4-e5f6-4071-9b08-000000000008 --output json", - "file": "88bb60105a.json" - }, - { - "match": "or jobs logs \"5b08c3d4-e5f6-4071-9b08-000000000008\" --level Error --output json", - "file": "31042d2c8d.json" - }, - { - "match": "or jobs logs 5b08c3d4-e5f6-4071-9b08-000000000008 --level Error --output json", - "file": "31042d2c8d.json" - }, - { - "match": "or jobs logs \"5b08c3d4-e5f6-4071-9b08-000000000008\" --output json", - "file": "e6645b8fc4.json" - }, - { - "match": "or jobs logs 5b08c3d4-e5f6-4071-9b08-000000000008 --output json", - "file": "e6645b8fc4.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/uip-fixture.json new file mode 100644 index 0000000000..48b2c52a11 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a08a2b3-d4e5-4f60-8a08-000000000008", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b08c3d4-e5f6-4071-9b08-000000000008\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraCommentSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T14:02:02.190Z\",\r\n \"EndTime\": \"2026-06-17T14:02:03.940Z\",\r\n \"CreationTime\": \"2026-06-17T14:02:01.760Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b08c3d4-e5f6-4071-9b08-000000000008", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.810Z\",\r\n \"Message\": \"[Add Comment] Response status code does not indicate success: 403 (Forbidden). You do not have permission to comment on this issue.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.880Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: You do not have permission to comment on this issue.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b08c3d4-e5f6-4071-9b08-000000000008", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b08c3d4-e5f6-4071-9b08-000000000008\",\r\n \"StartTime\": \"2026-06-17T14:02:02.190Z\",\r\n \"EndTime\": \"2026-06-17T14:02:03.940Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c08d4e5-f6a7-4182-9c08-000000000008\",\r\n \"Info\": \"Add Comment: Response status code does not indicate success: 403 (Forbidden). You do not have permission to comment on this issue.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: You do not have permission to comment on this issue.\",\r\n \"CreationTime\": \"2026-06-17T14:02:01.760Z\",\r\n \"ReleaseName\": \"JiraCommentSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013308,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a08a2b3-d4e5-4f60-8a08-000000000008\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T14:02:03.940Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b08c3d4-e5f6-4071-9b08-000000000008\",\r\n \"Id\": 813308208\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b08c3d4-e5f6-4071-9b08-000000000008", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-17T14:02:02.380Z\",\r\n \"Message\": \"JiraCommentSync execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.090Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.560Z\",\r\n \"Message\": \"Add Comment: posting comment to issue OPS-1502 as rpa-bot@company.com\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.810Z\",\r\n \"Message\": \"[Add Comment] Response status code does not indicate success: 403 (Forbidden). You do not have permission to comment on this issue.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.880Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: You do not have permission to comment on this issue.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-17T14:02:03.940Z\",\r\n \"Message\": \"JiraCommentSync execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a08a2b3-d4e5-4f60-8a08-000000000008\",\r\n \"Name\": \"Issue Sync\",\r\n \"Path\": \"Issue Sync\",\r\n \"Description\": \"Production folder for Jira commenting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/task.yaml index 1fa3cee33b..2fc2686d6b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-403-permission/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/README.md new file mode 100644 index 0000000000..70fb16318f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/README.md @@ -0,0 +1,46 @@ +# Add Comment Fails - Issue Not Found / IssueKey Missing Project Prefix (C2) + +This scenario reproduces an `Add Comment` failure caused by a malformed +`IssueKey`. The activity passes `1450` (a bare number) instead of the full Jira +key `OPS-1450`, so Jira returns `404 (Not Found) - Issue Not Found` and the +comment is never posted. + +## What this scenario uncovers + +**Root Cause:** `Add Comment` has `IssueKey="1450"` - missing the `PROJECT-` +prefix. Jira keys are `PROJECT-NNN`; a bare number does not resolve to an issue. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-add-comment-failures.md` +sub-case **C2** (Issue Not Found / bad IssueKey format). + +The scope authenticates fine (so this is **not** an auth failure), the status is +`404` rather than `403` (**not** the C3 permission case), and no `Visibility` is +configured (**not** C4). The user confirms the issue exists (browser URL +`.../browse/OPS-1450`), isolating the cause to the prefix-less key. The user is +framed as **off-host**, so the correct agent behavior is to tie the 404 to the +malformed `IssueKey` and recommend passing the full key - not to attempt host +commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Add Comment` using `IssueKey="1450"` (no project prefix) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-add-comment-failures.md` (sub-case C2) +- Agent identified the malformed `IssueKey` (missing the `PROJECT-` prefix) as + the cause (not an auth failure, a 403 permission problem, or a visibility + issue) and recommended passing the full issue key including the prefix, without + fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/24355d481a.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/24355d481a.json deleted file mode 100644 index 702a8824d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/24355d481a.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-17T10:18:02.520Z", - "Message": "JiraCommentBot execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T10:18:03.260Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T10:18:03.700Z", - "Message": "Add Comment: posting comment to issue 1450" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T10:18:03.920Z", - "Message": "[Add Comment] Response status code does not indicate success: 404 (Not Found). Issue Not Found. The issue '1450' does not exist or you do not have permission to view it." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T10:18:03.990Z", - "Message": "UiPath.Jira.Activities.JiraException: Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-17T10:18:04.050Z", - "Message": "JiraCommentBot execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/30d61c0ac1.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/30d61c0ac1.json deleted file mode 100644 index 086e9224ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/30d61c0ac1.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b07c3d4-e5f6-4071-9b07-000000000007", - "StartTime": "2026-06-17T10:18:02.330Z", - "EndTime": "2026-06-17T10:18:04.050Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c07d4e5-f6a7-4182-9c07-000000000007", - "Info": "Add Comment: Response status code does not indicate success: 404 (Not Found). Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.", - "CreationTime": "2026-06-17T10:18:01.880Z", - "ReleaseName": "JiraCommentBot", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013307, - "OrganizationUnitId": 7791618, - "FolderKey": "5a07a2b3-d4e5-4f60-8a07-000000000007", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T10:18:04.050Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b07c3d4-e5f6-4071-9b07-000000000007", - "Id": 813307207 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/336ee9b34c.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/336ee9b34c.json deleted file mode 100644 index 69811a3813..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/336ee9b34c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b07c3d4-e5f6-4071-9b07-000000000007", - "State": "Faulted", - "ProcessName": "JiraCommentBot", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-17T10:18:02.330Z", - "EndTime": "2026-06-17T10:18:04.050Z", - "CreationTime": "2026-06-17T10:18:01.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/46ac4679cc.json deleted file mode 100644 index 2afe1d0061..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a07a2b3-d4e5-4f60-8a07-000000000007", - "Name": "Issue Sync", - "Path": "Issue Sync", - "Description": "Production folder for Jira commenting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/README.md deleted file mode 100644 index d56106a6c9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Add Comment Fails - Issue Not Found / IssueKey Missing Project Prefix (C2) - -This scenario reproduces an `Add Comment` failure caused by a malformed -`IssueKey`. The activity passes `1450` (a bare number) instead of the full Jira -key `OPS-1450`, so Jira returns `404 (Not Found) - Issue Not Found` and the -comment is never posted. - -## What this scenario uncovers - -**Root Cause:** `Add Comment` has `IssueKey="1450"` - missing the `PROJECT-` -prefix. Jira keys are `PROJECT-NNN`; a bare number does not resolve to an issue. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-add-comment-failures.md` -sub-case **C2** (Issue Not Found / bad IssueKey format). - -The scope authenticates fine (so this is **not** an auth failure), the status is -`404` rather than `403` (**not** the C3 permission case), and no `Visibility` is -configured (**not** C4). The user confirms the issue exists (browser URL -`.../browse/OPS-1450`), isolating the cause to the prefix-less key. The user is -framed as **off-host**, so the correct agent behavior is to tie the 404 to the -malformed `IssueKey` and recommend passing the full key - not to attempt host -commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Add Comment` using `IssueKey="1450"` (no project prefix) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-add-comment-failures.md` (sub-case C2) -- Agent identified the malformed `IssueKey` (missing the `PROJECT-` prefix) as - the cause (not an auth failure, a 403 permission problem, or a visibility - issue) and recommended passing the full issue key including the prefix, without - fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/f4da4b6aa6.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/f4da4b6aa6.json deleted file mode 100644 index cbfabf1b55..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/f4da4b6aa6.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T10:18:03.920Z", - "Message": "[Add Comment] Response status code does not indicate success: 404 (Not Found). Issue Not Found. The issue '1450' does not exist or you do not have permission to view it." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T10:18:03.990Z", - "Message": "UiPath.Jira.Activities.JiraException: Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/manifest.json deleted file mode 100644 index 5aeb8dc3b4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a07a2b3-d4e5-4f60-8a07-000000000007\" --state Faulted --output json", - "file": "336ee9b34c.json" - }, - { - "match": "or jobs list --folder-key 5a07a2b3-d4e5-4f60-8a07-000000000007 --state Faulted --output json", - "file": "336ee9b34c.json" - }, - { - "match": "or jobs get \"5b07c3d4-e5f6-4071-9b07-000000000007\" --output json", - "file": "30d61c0ac1.json" - }, - { - "match": "or jobs get 5b07c3d4-e5f6-4071-9b07-000000000007 --output json", - "file": "30d61c0ac1.json" - }, - { - "match": "or jobs logs \"5b07c3d4-e5f6-4071-9b07-000000000007\" --level Error --output json", - "file": "f4da4b6aa6.json" - }, - { - "match": "or jobs logs 5b07c3d4-e5f6-4071-9b07-000000000007 --level Error --output json", - "file": "f4da4b6aa6.json" - }, - { - "match": "or jobs logs \"5b07c3d4-e5f6-4071-9b07-000000000007\" --output json", - "file": "24355d481a.json" - }, - { - "match": "or jobs logs 5b07c3d4-e5f6-4071-9b07-000000000007 --output json", - "file": "24355d481a.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..bdade1140e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a07a2b3-d4e5-4f60-8a07-000000000007", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b07c3d4-e5f6-4071-9b07-000000000007\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraCommentBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T10:18:02.330Z\",\r\n \"EndTime\": \"2026-06-17T10:18:04.050Z\",\r\n \"CreationTime\": \"2026-06-17T10:18:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b07c3d4-e5f6-4071-9b07-000000000007", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T10:18:03.920Z\",\r\n \"Message\": \"[Add Comment] Response status code does not indicate success: 404 (Not Found). Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T10:18:03.990Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b07c3d4-e5f6-4071-9b07-000000000007", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b07c3d4-e5f6-4071-9b07-000000000007\",\r\n \"StartTime\": \"2026-06-17T10:18:02.330Z\",\r\n \"EndTime\": \"2026-06-17T10:18:04.050Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c07d4e5-f6a7-4182-9c07-000000000007\",\r\n \"Info\": \"Add Comment: Response status code does not indicate success: 404 (Not Found). Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\",\r\n \"CreationTime\": \"2026-06-17T10:18:01.880Z\",\r\n \"ReleaseName\": \"JiraCommentBot\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013307,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a07a2b3-d4e5-4f60-8a07-000000000007\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T10:18:04.050Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b07c3d4-e5f6-4071-9b07-000000000007\",\r\n \"Id\": 813307207\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b07c3d4-e5f6-4071-9b07-000000000007", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-17T10:18:02.520Z\",\r\n \"Message\": \"JiraCommentBot execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T10:18:03.260Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T10:18:03.700Z\",\r\n \"Message\": \"Add Comment: posting comment to issue 1450\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T10:18:03.920Z\",\r\n \"Message\": \"[Add Comment] Response status code does not indicate success: 404 (Not Found). Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T10:18:03.990Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Issue Not Found. The issue '1450' does not exist or you do not have permission to view it.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-17T10:18:04.050Z\",\r\n \"Message\": \"JiraCommentBot execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a07a2b3-d4e5-4f60-8a07-000000000007\",\r\n \"Name\": \"Issue Sync\",\r\n \"Path\": \"Issue Sync\",\r\n \"Description\": \"Production folder for Jira commenting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/task.yaml index 1d37a8843a..4f702cbca6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-issue-not-found/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/README.md new file mode 100644 index 0000000000..8f1bc294d9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/README.md @@ -0,0 +1,49 @@ +# Add Comment Fails - Comment Visibility Restriction (C4) + +This scenario reproduces an `Add Comment` failure caused by a misconfigured +comment `Visibility`. The activity restricts the comment to the project role +`Service Desk Team`, which the bot account is not a member of, so Jira rejects it +with `400 (Bad Request): You are currently not a member of the project role +'Service Desk Team' that you are restricting the comment visibility to`. + +## What this scenario uncovers + +**Root Cause:** `Add Comment` sets `VisibilityType="Role"` / +`VisibilityValue="Service Desk Team"`. Jira requires the commenting account to +belong to the role/group a restricted comment targets; the bot account does not, +and the user only wants a public comment. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-add-comment-failures.md` +sub-case **C4** (comment Visibility restriction). + +The scope authenticates fine (so this is **not** an auth failure), the status is +`400` about a visibility role rather than `404` (**not** the C2 bad-key case) or a +blanket `403` (**not** the C3 missing-permission case). The user confirms a normal +public comment is fine. The user is framed as **off-host**, so the correct agent +behavior is to tie the rejection to the `Visibility` restriction and recommend +clearing it - not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Add Comment` that sets `VisibilityType="Role"` / `VisibilityValue="Service Desk Team"` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-add-comment-failures.md` (sub-case C4) +- Agent identified the misconfigured comment `Visibility` restriction (a project + role the account is not in) as the cause (not an auth failure, a bad IssueKey, + or a missing Add Comments permission) and recommended clearing `Visibility` for + a public comment (or setting a role/group the account belongs to), without + fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/46ac4679cc.json deleted file mode 100644 index 9f97e83044..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a09a2b3-d4e5-4f60-8a09-000000000009", - "Name": "Service Desk", - "Path": "Service Desk", - "Description": "Production folder for Jira commenting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/5cc74fb37c.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/5cc74fb37c.json deleted file mode 100644 index e9b25ea8aa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/5cc74fb37c.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b09c3d4-e5f6-4071-9b09-000000000009", - "StartTime": "2026-06-17T16:25:02.270Z", - "EndTime": "2026-06-17T16:25:04.020Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c09d4e5-f6a7-4182-9c09-000000000009", - "Info": "Add Comment: Response status code does not indicate success: 400 (Bad Request). You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.", - "CreationTime": "2026-06-17T16:25:01.820Z", - "ReleaseName": "JiraNoteBot", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013309, - "OrganizationUnitId": 7791618, - "FolderKey": "5a09a2b3-d4e5-4f60-8a09-000000000009", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T16:25:04.020Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b09c3d4-e5f6-4071-9b09-000000000009", - "Id": 813309209 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/62d8e0105e.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/62d8e0105e.json deleted file mode 100644 index 44f4d03ad5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/62d8e0105e.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T16:25:03.890Z", - "Message": "[Add Comment] Response status code does not indicate success: 400 (Bad Request). You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T16:25:03.960Z", - "Message": "UiPath.Jira.Activities.JiraException: You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/README.md deleted file mode 100644 index 78f726ae2a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Add Comment Fails - Comment Visibility Restriction (C4) - -This scenario reproduces an `Add Comment` failure caused by a misconfigured -comment `Visibility`. The activity restricts the comment to the project role -`Service Desk Team`, which the bot account is not a member of, so Jira rejects it -with `400 (Bad Request): You are currently not a member of the project role -'Service Desk Team' that you are restricting the comment visibility to`. - -## What this scenario uncovers - -**Root Cause:** `Add Comment` sets `VisibilityType="Role"` / -`VisibilityValue="Service Desk Team"`. Jira requires the commenting account to -belong to the role/group a restricted comment targets; the bot account does not, -and the user only wants a public comment. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-add-comment-failures.md` -sub-case **C4** (comment Visibility restriction). - -The scope authenticates fine (so this is **not** an auth failure), the status is -`400` about a visibility role rather than `404` (**not** the C2 bad-key case) or a -blanket `403` (**not** the C3 missing-permission case). The user confirms a normal -public comment is fine. The user is framed as **off-host**, so the correct agent -behavior is to tie the rejection to the `Visibility` restriction and recommend -clearing it - not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Add Comment` that sets `VisibilityType="Role"` / `VisibilityValue="Service Desk Team"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-add-comment-failures.md` (sub-case C4) -- Agent identified the misconfigured comment `Visibility` restriction (a project - role the account is not in) as the cause (not an auth failure, a bad IssueKey, - or a missing Add Comments permission) and recommended clearing `Visibility` for - a public comment (or setting a role/group the account belongs to), without - fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/a686a9e6df.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/a686a9e6df.json deleted file mode 100644 index f67d76ed10..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/a686a9e6df.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-17T16:25:02.460Z", - "Message": "JiraNoteBot execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T16:25:03.170Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T16:25:03.640Z", - "Message": "Add Comment: posting comment to issue OPS-1605 (visibility: Role 'Service Desk Team')" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T16:25:03.890Z", - "Message": "[Add Comment] Response status code does not indicate success: 400 (Bad Request). You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T16:25:03.960Z", - "Message": "UiPath.Jira.Activities.JiraException: You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at AddComment \"Add Comment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-17T16:25:04.020Z", - "Message": "JiraNoteBot execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/c4ca569ecd.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/c4ca569ecd.json deleted file mode 100644 index 70a7a463da..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/c4ca569ecd.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b09c3d4-e5f6-4071-9b09-000000000009", - "State": "Faulted", - "ProcessName": "JiraNoteBot", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-17T16:25:02.270Z", - "EndTime": "2026-06-17T16:25:04.020Z", - "CreationTime": "2026-06-17T16:25:01.820Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/manifest.json deleted file mode 100644 index ba1f0bebb6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a09a2b3-d4e5-4f60-8a09-000000000009\" --state Faulted --output json", - "file": "c4ca569ecd.json" - }, - { - "match": "or jobs list --folder-key 5a09a2b3-d4e5-4f60-8a09-000000000009 --state Faulted --output json", - "file": "c4ca569ecd.json" - }, - { - "match": "or jobs get \"5b09c3d4-e5f6-4071-9b09-000000000009\" --output json", - "file": "5cc74fb37c.json" - }, - { - "match": "or jobs get 5b09c3d4-e5f6-4071-9b09-000000000009 --output json", - "file": "5cc74fb37c.json" - }, - { - "match": "or jobs logs \"5b09c3d4-e5f6-4071-9b09-000000000009\" --level Error --output json", - "file": "62d8e0105e.json" - }, - { - "match": "or jobs logs 5b09c3d4-e5f6-4071-9b09-000000000009 --level Error --output json", - "file": "62d8e0105e.json" - }, - { - "match": "or jobs logs \"5b09c3d4-e5f6-4071-9b09-000000000009\" --output json", - "file": "a686a9e6df.json" - }, - { - "match": "or jobs logs 5b09c3d4-e5f6-4071-9b09-000000000009 --output json", - "file": "a686a9e6df.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/uip-fixture.json new file mode 100644 index 0000000000..3c10d58b9b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a09a2b3-d4e5-4f60-8a09-000000000009", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b09c3d4-e5f6-4071-9b09-000000000009\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraNoteBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T16:25:02.270Z\",\r\n \"EndTime\": \"2026-06-17T16:25:04.020Z\",\r\n \"CreationTime\": \"2026-06-17T16:25:01.820Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b09c3d4-e5f6-4071-9b09-000000000009", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T16:25:03.890Z\",\r\n \"Message\": \"[Add Comment] Response status code does not indicate success: 400 (Bad Request). You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T16:25:03.960Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b09c3d4-e5f6-4071-9b09-000000000009", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b09c3d4-e5f6-4071-9b09-000000000009\",\r\n \"StartTime\": \"2026-06-17T16:25:02.270Z\",\r\n \"EndTime\": \"2026-06-17T16:25:04.020Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c09d4e5-f6a7-4182-9c09-000000000009\",\r\n \"Info\": \"Add Comment: Response status code does not indicate success: 400 (Bad Request). You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\",\r\n \"CreationTime\": \"2026-06-17T16:25:01.820Z\",\r\n \"ReleaseName\": \"JiraNoteBot\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013309,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a09a2b3-d4e5-4f60-8a09-000000000009\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T16:25:04.020Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b09c3d4-e5f6-4071-9b09-000000000009\",\r\n \"Id\": 813309209\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b09c3d4-e5f6-4071-9b09-000000000009", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-17T16:25:02.460Z\",\r\n \"Message\": \"JiraNoteBot execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T16:25:03.170Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T16:25:03.640Z\",\r\n \"Message\": \"Add Comment: posting comment to issue OPS-1605 (visibility: Role 'Service Desk Team')\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T16:25:03.890Z\",\r\n \"Message\": \"[Add Comment] Response status code does not indicate success: 400 (Bad Request). You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T16:25:03.960Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: You are currently not a member of the project role 'Service Desk Team' that you are restricting the comment visibility to.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at AddComment \\\"Add Comment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-17T16:25:04.020Z\",\r\n \"Message\": \"JiraNoteBot execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a09a2b3-d4e5-4f60-8a09-000000000009\",\r\n \"Name\": \"Service Desk\",\r\n \"Path\": \"Service Desk\",\r\n \"Description\": \"Production folder for Jira commenting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/task.yaml index dfdd9118a1..c7d4c76cb8 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-comment-visibility-restriction/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/README.md new file mode 100644 index 0000000000..d69db3174f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/README.md @@ -0,0 +1,48 @@ +# Jira Response Was Not Recognized As JSON - Server URL Truncation (R1) + +This scenario reproduces a `Jira Scope` child-activity failure caused by a +`Server URL` that points past the instance root. The classic +`UiPath.Jira.Activities` pack appends its own `/rest/api/...` paths to +`Server URL`; with `/secure/Dashboard.jspa` (a Jira web UI page) appended, the +`Search Issues` REST call hits an HTML page and the pack fails with `Response was +not recognized as JSON`. + +## What this scenario uncovers + +**Root Cause:** The `Jira Scope` `Server URL` is +`https://acme.atlassian.net/secure/Dashboard.jspa` instead of the bare root +`https://acme.atlassian.net`. The scope authenticates fine; the child REST call +is mis-routed to an HTML page. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-scope-response-not-json-or-500.md` +sub-case **R1** (URL truncation - path appended past the root instance). + +The logs show the session opened successfully (so this is **not** an +authentication failure), and the host is `*.atlassian.net` (Jira Cloud, so **not** +the on-prem Server / Data Center sub-case R2). The user is framed as **off-host**, +so the correct agent behavior is to tie the parse failure to the appended URL +path and recommend the root URL - not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` (Server URL with `/secure/Dashboard.jspa` appended) wrapping `Search Issues` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-scope-response-not-json-or-500.md` (sub-case R1) +- Agent identified the `Server URL` truncation (a dashboard/page path appended + past the root instance) as the cause - not an auth failure, not an on-prem + endpoint issue - and recommended setting `Server URL` to the bare root + `https://acme.atlassian.net`, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/1aeff360bf.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/1aeff360bf.json deleted file mode 100644 index 7bf2d6f9a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/1aeff360bf.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-15T06:00:02.520Z", - "Message": "JiraWeeklyReport execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T06:00:03.610Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net/secure/Dashboard.jspa (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T06:00:04.730Z", - "Message": "Search Issues: sending request (JQL: project = OPS AND created >= -7d)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T06:00:04.980Z", - "Message": "[Search Issues] Response was not recognized as JSON." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T06:00:05.060Z", - "Message": "UiPath.Jira.Activities.JiraException: Response was not recognized as JSON.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at SearchIssues \"Search Issues\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-15T06:00:05.110Z", - "Message": "JiraWeeklyReport execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/282666ec20.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/282666ec20.json deleted file mode 100644 index 300bfda9d5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/282666ec20.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b02c3d4-e5f6-4071-9b02-000000000002", - "StartTime": "2026-06-15T06:00:02.340Z", - "EndTime": "2026-06-15T06:00:05.110Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c02d4e5-f6a7-4182-9c02-000000000002", - "Info": "Search Issues: Response was not recognized as JSON.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at SearchIssues \"Search Issues\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: Response was not recognized as JSON.", - "CreationTime": "2026-06-15T06:00:01.900Z", - "ReleaseName": "JiraWeeklyReport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013302, - "OrganizationUnitId": 7791618, - "FolderKey": "5a02a2b3-d4e5-4f60-8a02-000000000002", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T06:00:05.110Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b02c3d4-e5f6-4071-9b02-000000000002", - "Id": 813302202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/46ac4679cc.json deleted file mode 100644 index 7df2150e0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a02a2b3-d4e5-4f60-8a02-000000000002", - "Name": "Reporting", - "Path": "Reporting", - "Description": "Production folder for Jira reporting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/5a4a4f1dd8.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/5a4a4f1dd8.json deleted file mode 100644 index 3389e981b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/5a4a4f1dd8.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b02c3d4-e5f6-4071-9b02-000000000002", - "State": "Faulted", - "ProcessName": "JiraWeeklyReport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-15T06:00:02.340Z", - "EndTime": "2026-06-15T06:00:05.110Z", - "CreationTime": "2026-06-15T06:00:01.900Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/68cecd381e.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/68cecd381e.json deleted file mode 100644 index a2a8d2266e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/68cecd381e.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-15T06:00:04.980Z", - "Message": "[Search Issues] Response was not recognized as JSON." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-15T06:00:05.060Z", - "Message": "UiPath.Jira.Activities.JiraException: Response was not recognized as JSON.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at SearchIssues \"Search Issues\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/README.md deleted file mode 100644 index b205c2ac38..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Jira Response Was Not Recognized As JSON - Server URL Truncation (R1) - -This scenario reproduces a `Jira Scope` child-activity failure caused by a -`Server URL` that points past the instance root. The classic -`UiPath.Jira.Activities` pack appends its own `/rest/api/...` paths to -`Server URL`; with `/secure/Dashboard.jspa` (a Jira web UI page) appended, the -`Search Issues` REST call hits an HTML page and the pack fails with `Response was -not recognized as JSON`. - -## What this scenario uncovers - -**Root Cause:** The `Jira Scope` `Server URL` is -`https://acme.atlassian.net/secure/Dashboard.jspa` instead of the bare root -`https://acme.atlassian.net`. The scope authenticates fine; the child REST call -is mis-routed to an HTML page. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-scope-response-not-json-or-500.md` -sub-case **R1** (URL truncation - path appended past the root instance). - -The logs show the session opened successfully (so this is **not** an -authentication failure), and the host is `*.atlassian.net` (Jira Cloud, so **not** -the on-prem Server / Data Center sub-case R2). The user is framed as **off-host**, -so the correct agent behavior is to tie the parse failure to the appended URL -path and recommend the root URL - not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` (Server URL with `/secure/Dashboard.jspa` appended) wrapping `Search Issues` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-scope-response-not-json-or-500.md` (sub-case R1) -- Agent identified the `Server URL` truncation (a dashboard/page path appended - past the root instance) as the cause - not an auth failure, not an on-prem - endpoint issue - and recommended setting `Server URL` to the bare root - `https://acme.atlassian.net`, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/manifest.json deleted file mode 100644 index a003a64c9d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a02a2b3-d4e5-4f60-8a02-000000000002\" --state Faulted --output json", - "file": "5a4a4f1dd8.json" - }, - { - "match": "or jobs list --folder-key 5a02a2b3-d4e5-4f60-8a02-000000000002 --state Faulted --output json", - "file": "5a4a4f1dd8.json" - }, - { - "match": "or jobs get \"5b02c3d4-e5f6-4071-9b02-000000000002\" --output json", - "file": "282666ec20.json" - }, - { - "match": "or jobs get 5b02c3d4-e5f6-4071-9b02-000000000002 --output json", - "file": "282666ec20.json" - }, - { - "match": "or jobs logs \"5b02c3d4-e5f6-4071-9b02-000000000002\" --level Error --output json", - "file": "68cecd381e.json" - }, - { - "match": "or jobs logs 5b02c3d4-e5f6-4071-9b02-000000000002 --level Error --output json", - "file": "68cecd381e.json" - }, - { - "match": "or jobs logs \"5b02c3d4-e5f6-4071-9b02-000000000002\" --output json", - "file": "1aeff360bf.json" - }, - { - "match": "or jobs logs 5b02c3d4-e5f6-4071-9b02-000000000002 --output json", - "file": "1aeff360bf.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/uip-fixture.json new file mode 100644 index 0000000000..c90ba2323b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a02a2b3-d4e5-4f60-8a02-000000000002", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b02c3d4-e5f6-4071-9b02-000000000002\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraWeeklyReport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T06:00:02.340Z\",\r\n \"EndTime\": \"2026-06-15T06:00:05.110Z\",\r\n \"CreationTime\": \"2026-06-15T06:00:01.900Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b02c3d4-e5f6-4071-9b02-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T06:00:04.980Z\",\r\n \"Message\": \"[Search Issues] Response was not recognized as JSON.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T06:00:05.060Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Response was not recognized as JSON.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at SearchIssues \\\"Search Issues\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b02c3d4-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b02c3d4-e5f6-4071-9b02-000000000002\",\r\n \"StartTime\": \"2026-06-15T06:00:02.340Z\",\r\n \"EndTime\": \"2026-06-15T06:00:05.110Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c02d4e5-f6a7-4182-9c02-000000000002\",\r\n \"Info\": \"Search Issues: Response was not recognized as JSON.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at SearchIssues \\\"Search Issues\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: Response was not recognized as JSON.\",\r\n \"CreationTime\": \"2026-06-15T06:00:01.900Z\",\r\n \"ReleaseName\": \"JiraWeeklyReport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013302,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a02a2b3-d4e5-4f60-8a02-000000000002\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T06:00:05.110Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b02c3d4-e5f6-4071-9b02-000000000002\",\r\n \"Id\": 813302202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b02c3d4-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-15T06:00:02.520Z\",\r\n \"Message\": \"JiraWeeklyReport execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T06:00:03.610Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net/secure/Dashboard.jspa (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T06:00:04.730Z\",\r\n \"Message\": \"Search Issues: sending request (JQL: project = OPS AND created >= -7d)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T06:00:04.980Z\",\r\n \"Message\": \"[Search Issues] Response was not recognized as JSON.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-15T06:00:05.060Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Response was not recognized as JSON.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at SearchIssues \\\"Search Issues\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-15T06:00:05.110Z\",\r\n \"Message\": \"JiraWeeklyReport execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a02a2b3-d4e5-4f60-8a02-000000000002\",\r\n \"Name\": \"Reporting\",\r\n \"Path\": \"Reporting\",\r\n \"Description\": \"Production folder for Jira reporting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/task.yaml index fa8ac59bcf..df068bb710 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-response-not-json-500/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/README.md new file mode 100644 index 0000000000..a1baad20db --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/README.md @@ -0,0 +1,53 @@ +# Transition Workflow Fails - Metadata Deserialization Bug in an Old Pack (T4) + +This scenario reproduces a Jira transition workflow that faults with a +`Newtonsoft.Json.JsonSerializationException: Error converting value "add" to type +'Atlassian.Jira.IssueFieldEditMetadataOperation'` while `Get Transitions` parses +the transition field metadata. The project pins an **old** +`UiPath.Jira.Activities` version (`1.5.0`) with a known metadata-deserialization +bug on complex fields (here a custom field with an `operations` array). + +## What this scenario uncovers + +**Root Cause:** A version-bound deserialization bug. The workflow is configured +correctly (dynamic `Get Transitions`, valid auth/URL), but `UiPath.Jira.Activities +[1.5.0]` cannot map the complex field metadata to its +`Atlassian.Jira.IssueFieldEditMetadataOperation` type. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-transition-issue-failures.md` +sub-case **T4** (deserialization / package-version type mismatch; upgrade the +pack or migrate to the Integration Service connector). + +The scope authenticates fine (so this is **not** an auth failure), the error is a +JSON type-conversion exception rather than `Field '' is required` (**not** +T1) or `Transition '' is not valid` (**not** T2), and it is not a +rule/permission denial (**not** T3). The workflow even uses the correct dynamic +`Get Transitions` pattern, isolating the cause to the package version. The user +is framed as **off-host**, so the correct agent behavior is to tie the +deserialization error to the old pack version and recommend upgrading (or +migrating to Integration Service) - not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project pinning `UiPath.Jira.Activities [1.5.0]`, with a `Jira Scope` -> `Get Transitions` -> `Transition Issue` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-transition-issue-failures.md` (sub-case T4) +- Agent identified the old-package metadata-deserialization bug + (`IssueFieldEditMetadataOperation`) as the cause (not a missing field, invalid + ID, or permission/validator) and recommended upgrading `UiPath.Jira.Activities` + to a current release or migrating to the Integration Service Jira connector, + without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/2ed06815de.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/2ed06815de.json deleted file mode 100644 index 6cbc96d747..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/2ed06815de.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T17:22:03.880Z", - "Message": "[Get Transitions] Error converting value \"add\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'. Path 'fields.customfield_10044.operations[0]', line 1, position 842." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T17:22:03.950Z", - "Message": "Newtonsoft.Json.JsonSerializationException: Error converting value \"add\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at GetTransitions \"Get Transitions\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/46ac4679cc.json deleted file mode 100644 index d57d806258..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a06a2b3-d4e5-4f60-8a06-000000000006", - "Name": "Escalations", - "Path": "Escalations", - "Description": "Production folder for Jira escalation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/58087e9140.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/58087e9140.json deleted file mode 100644 index 0400384ca5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/58087e9140.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b06c3d4-e5f6-4071-9b06-000000000006", - "State": "Faulted", - "ProcessName": "JiraEscalation", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T17:22:02.260Z", - "EndTime": "2026-06-16T17:22:04.010Z", - "CreationTime": "2026-06-16T17:22:01.820Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/60049b5831.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/60049b5831.json deleted file mode 100644 index 920b16de3a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/60049b5831.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-16T17:22:02.450Z", - "Message": "JiraEscalation execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T17:22:03.210Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T17:22:03.640Z", - "Message": "Get Transitions: fetching transitions with field metadata for issue OPS-2087" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T17:22:03.880Z", - "Message": "[Get Transitions] Error converting value \"add\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'. Path 'fields.customfield_10044.operations[0]', line 1, position 842." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T17:22:03.950Z", - "Message": "Newtonsoft.Json.JsonSerializationException: Error converting value \"add\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at GetTransitions \"Get Transitions\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-16T17:22:04.010Z", - "Message": "JiraEscalation execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/README.md deleted file mode 100644 index 7a398feefa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# Transition Workflow Fails - Metadata Deserialization Bug in an Old Pack (T4) - -This scenario reproduces a Jira transition workflow that faults with a -`Newtonsoft.Json.JsonSerializationException: Error converting value "add" to type -'Atlassian.Jira.IssueFieldEditMetadataOperation'` while `Get Transitions` parses -the transition field metadata. The project pins an **old** -`UiPath.Jira.Activities` version (`1.5.0`) with a known metadata-deserialization -bug on complex fields (here a custom field with an `operations` array). - -## What this scenario uncovers - -**Root Cause:** A version-bound deserialization bug. The workflow is configured -correctly (dynamic `Get Transitions`, valid auth/URL), but `UiPath.Jira.Activities -[1.5.0]` cannot map the complex field metadata to its -`Atlassian.Jira.IssueFieldEditMetadataOperation` type. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-transition-issue-failures.md` -sub-case **T4** (deserialization / package-version type mismatch; upgrade the -pack or migrate to the Integration Service connector). - -The scope authenticates fine (so this is **not** an auth failure), the error is a -JSON type-conversion exception rather than `Field '' is required` (**not** -T1) or `Transition '' is not valid` (**not** T2), and it is not a -rule/permission denial (**not** T3). The workflow even uses the correct dynamic -`Get Transitions` pattern, isolating the cause to the package version. The user -is framed as **off-host**, so the correct agent behavior is to tie the -deserialization error to the old pack version and recommend upgrading (or -migrating to Integration Service) - not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project pinning `UiPath.Jira.Activities [1.5.0]`, with a `Jira Scope` -> `Get Transitions` -> `Transition Issue` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-transition-issue-failures.md` (sub-case T4) -- Agent identified the old-package metadata-deserialization bug - (`IssueFieldEditMetadataOperation`) as the cause (not a missing field, invalid - ID, or permission/validator) and recommended upgrading `UiPath.Jira.Activities` - to a current release or migrating to the Integration Service Jira connector, - without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/bbf7763e85.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/bbf7763e85.json deleted file mode 100644 index 3df0547556..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/bbf7763e85.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b06c3d4-e5f6-4071-9b06-000000000006", - "StartTime": "2026-06-16T17:22:02.260Z", - "EndTime": "2026-06-16T17:22:04.010Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c06d4e5-f6a7-4182-9c06-000000000006", - "Info": "Get Transitions: Error converting value \"add\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'. Path 'fields.customfield_10044.operations[0]', line 1, position 842.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at GetTransitions \"Get Transitions\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nNewtonsoft.Json.JsonSerializationException: Error converting value \"add\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'.", - "CreationTime": "2026-06-16T17:22:01.820Z", - "ReleaseName": "JiraEscalation", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013306, - "OrganizationUnitId": 7791618, - "FolderKey": "5a06a2b3-d4e5-4f60-8a06-000000000006", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T17:22:04.010Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b06c3d4-e5f6-4071-9b06-000000000006", - "Id": 813306206 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/manifest.json deleted file mode 100644 index 7ce643f38b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a06a2b3-d4e5-4f60-8a06-000000000006\" --state Faulted --output json", - "file": "58087e9140.json" - }, - { - "match": "or jobs list --folder-key 5a06a2b3-d4e5-4f60-8a06-000000000006 --state Faulted --output json", - "file": "58087e9140.json" - }, - { - "match": "or jobs get \"5b06c3d4-e5f6-4071-9b06-000000000006\" --output json", - "file": "bbf7763e85.json" - }, - { - "match": "or jobs get 5b06c3d4-e5f6-4071-9b06-000000000006 --output json", - "file": "bbf7763e85.json" - }, - { - "match": "or jobs logs \"5b06c3d4-e5f6-4071-9b06-000000000006\" --level Error --output json", - "file": "2ed06815de.json" - }, - { - "match": "or jobs logs 5b06c3d4-e5f6-4071-9b06-000000000006 --level Error --output json", - "file": "2ed06815de.json" - }, - { - "match": "or jobs logs \"5b06c3d4-e5f6-4071-9b06-000000000006\" --output json", - "file": "60049b5831.json" - }, - { - "match": "or jobs logs 5b06c3d4-e5f6-4071-9b06-000000000006 --output json", - "file": "60049b5831.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/uip-fixture.json new file mode 100644 index 0000000000..450e940aed --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a06a2b3-d4e5-4f60-8a06-000000000006", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b06c3d4-e5f6-4071-9b06-000000000006\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraEscalation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T17:22:02.260Z\",\r\n \"EndTime\": \"2026-06-16T17:22:04.010Z\",\r\n \"CreationTime\": \"2026-06-16T17:22:01.820Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b06c3d4-e5f6-4071-9b06-000000000006", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T17:22:03.880Z\",\r\n \"Message\": \"[Get Transitions] Error converting value \\\"add\\\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'. Path 'fields.customfield_10044.operations[0]', line 1, position 842.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T17:22:03.950Z\",\r\n \"Message\": \"Newtonsoft.Json.JsonSerializationException: Error converting value \\\"add\\\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at GetTransitions \\\"Get Transitions\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b06c3d4-e5f6-4071-9b06-000000000006", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b06c3d4-e5f6-4071-9b06-000000000006\",\r\n \"StartTime\": \"2026-06-16T17:22:02.260Z\",\r\n \"EndTime\": \"2026-06-16T17:22:04.010Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c06d4e5-f6a7-4182-9c06-000000000006\",\r\n \"Info\": \"Get Transitions: Error converting value \\\"add\\\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'. Path 'fields.customfield_10044.operations[0]', line 1, position 842.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at GetTransitions \\\"Get Transitions\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nNewtonsoft.Json.JsonSerializationException: Error converting value \\\"add\\\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'.\",\r\n \"CreationTime\": \"2026-06-16T17:22:01.820Z\",\r\n \"ReleaseName\": \"JiraEscalation\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013306,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a06a2b3-d4e5-4f60-8a06-000000000006\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T17:22:04.010Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b06c3d4-e5f6-4071-9b06-000000000006\",\r\n \"Id\": 813306206\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b06c3d4-e5f6-4071-9b06-000000000006", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-16T17:22:02.450Z\",\r\n \"Message\": \"JiraEscalation execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T17:22:03.210Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T17:22:03.640Z\",\r\n \"Message\": \"Get Transitions: fetching transitions with field metadata for issue OPS-2087\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T17:22:03.880Z\",\r\n \"Message\": \"[Get Transitions] Error converting value \\\"add\\\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'. Path 'fields.customfield_10044.operations[0]', line 1, position 842.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T17:22:03.950Z\",\r\n \"Message\": \"Newtonsoft.Json.JsonSerializationException: Error converting value \\\"add\\\" to type 'Atlassian.Jira.IssueFieldEditMetadataOperation'.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at GetTransitions \\\"Get Transitions\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-16T17:22:04.010Z\",\r\n \"Message\": \"JiraEscalation execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a06a2b3-d4e5-4f60-8a06-000000000006\",\r\n \"Name\": \"Escalations\",\r\n \"Path\": \"Escalations\",\r\n \"Description\": \"Production folder for Jira escalation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/task.yaml index b9dcdca3d9..b9e6356b66 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-deserialization/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/README.md new file mode 100644 index 0000000000..b79f7ac1e2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/README.md @@ -0,0 +1,50 @@ +# Transition Issue Fails - Hardcoded / Invalid Transition ID (T2) + +This scenario reproduces a `Transition Issue` failure caused by a **hardcoded +transition ID** that is not legal from the ticket's current status. Jira +transition IDs are per-status edges, not global constants, so a literal +`TransitionId="31"` that worked from one status is rejected (`400 Bad Request: +Transition id '31' is not valid ...`) once a ticket starts from a different +status. + +## What this scenario uncovers + +**Root Cause:** `Transition Issue` uses a hardcoded `TransitionId="31"` with no +`Get Transitions` call. The failing issue (`OPS-1421`) is in status `In Review`, +which has no transition with ID 31. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-transition-issue-failures.md` +sub-case **T2** (invalid / outdated transition ID; resolve dynamically via Get +Transitions). + +The scope authenticates fine (so this is **not** an auth failure), the error is +`is not valid` rather than `is required` (**not** the T1 missing-field case), it +is a routing rejection rather than a rule/permission denial (**not** T3), and +there is no `IssueFieldEditMetadataOperation` conversion error (**not** T4). The +user is framed as **off-host**, so the correct agent behavior is to tie the +rejection to the hardcoded ID and recommend resolving it via `Get Transitions` - +not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Transition Issue` using a hardcoded `TransitionId="31"` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-transition-issue-failures.md` (sub-case T2) +- Agent identified the hardcoded/invalid transition ID for the issue's current + status as the cause (not a missing required field, permission/validator, or + deserialization bug) and recommended resolving the transition ID dynamically + via `Get Transitions`, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/46ac4679cc.json deleted file mode 100644 index 073bc9dbbc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a04a2b3-d4e5-4f60-8a04-000000000004", - "Name": "Issue Sync", - "Path": "Issue Sync", - "Description": "Production folder for Jira issue-sync processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/7a82fd653d.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/7a82fd653d.json deleted file mode 100644 index d23b645d9c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/7a82fd653d.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b04c3d4-e5f6-4071-9b04-000000000004", - "State": "Faulted", - "ProcessName": "JiraStatusUpdater", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T13:05:02.410Z", - "EndTime": "2026-06-16T13:05:04.220Z", - "CreationTime": "2026-06-16T13:05:01.950Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/9280daa270.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/9280daa270.json deleted file mode 100644 index b27a751af9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/9280daa270.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b04c3d4-e5f6-4071-9b04-000000000004", - "StartTime": "2026-06-16T13:05:02.410Z", - "EndTime": "2026-06-16T13:05:04.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c04d4e5-f6a7-4182-9c04-000000000004", - "Info": "Transition Issue: Response status code does not indicate success: 400 (Bad Request). Transition id '31' is not valid for issue OPS-1421 in its current status.\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at TransitionIssue \"Transition Issue\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: Transition id '31' is not valid for issue OPS-1421 in its current status.", - "CreationTime": "2026-06-16T13:05:01.950Z", - "ReleaseName": "JiraStatusUpdater", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013304, - "OrganizationUnitId": 7791618, - "FolderKey": "5a04a2b3-d4e5-4f60-8a04-000000000004", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T13:05:04.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b04c3d4-e5f6-4071-9b04-000000000004", - "Id": 813304204 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/README.md deleted file mode 100644 index 11e7f8d0d2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# Transition Issue Fails - Hardcoded / Invalid Transition ID (T2) - -This scenario reproduces a `Transition Issue` failure caused by a **hardcoded -transition ID** that is not legal from the ticket's current status. Jira -transition IDs are per-status edges, not global constants, so a literal -`TransitionId="31"` that worked from one status is rejected (`400 Bad Request: -Transition id '31' is not valid ...`) once a ticket starts from a different -status. - -## What this scenario uncovers - -**Root Cause:** `Transition Issue` uses a hardcoded `TransitionId="31"` with no -`Get Transitions` call. The failing issue (`OPS-1421`) is in status `In Review`, -which has no transition with ID 31. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-transition-issue-failures.md` -sub-case **T2** (invalid / outdated transition ID; resolve dynamically via Get -Transitions). - -The scope authenticates fine (so this is **not** an auth failure), the error is -`is not valid` rather than `is required` (**not** the T1 missing-field case), it -is a routing rejection rather than a rule/permission denial (**not** T3), and -there is no `IssueFieldEditMetadataOperation` conversion error (**not** T4). The -user is framed as **off-host**, so the correct agent behavior is to tie the -rejection to the hardcoded ID and recommend resolving it via `Get Transitions` - -not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Transition Issue` using a hardcoded `TransitionId="31"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-transition-issue-failures.md` (sub-case T2) -- Agent identified the hardcoded/invalid transition ID for the issue's current - status as the cause (not a missing required field, permission/validator, or - deserialization bug) and recommended resolving the transition ID dynamically - via `Get Transitions`, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/c95bb27fe0.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/c95bb27fe0.json deleted file mode 100644 index 0584954966..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/c95bb27fe0.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T13:05:04.090Z", - "Message": "[Transition Issue] Response status code does not indicate success: 400 (Bad Request). Transition id '31' is not valid for issue OPS-1421 in its current status." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T13:05:04.170Z", - "Message": "UiPath.Jira.Activities.JiraException: Transition id '31' is not valid for issue OPS-1421 in its current status.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at TransitionIssue \"Transition Issue\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/e00230a00f.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/e00230a00f.json deleted file mode 100644 index 23627313b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/e00230a00f.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-16T13:05:02.600Z", - "Message": "JiraStatusUpdater execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T13:05:03.380Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T13:05:03.910Z", - "Message": "Transition Issue: requesting transition id 31 on issue OPS-1421 (current status: In Review)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T13:05:04.090Z", - "Message": "[Transition Issue] Response status code does not indicate success: 400 (Bad Request). Transition id '31' is not valid for issue OPS-1421 in its current status." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T13:05:04.170Z", - "Message": "UiPath.Jira.Activities.JiraException: Transition id '31' is not valid for issue OPS-1421 in its current status.\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at TransitionIssue \"Transition Issue\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-16T13:05:04.220Z", - "Message": "JiraStatusUpdater execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/manifest.json deleted file mode 100644 index 7c3e4dad5c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a04a2b3-d4e5-4f60-8a04-000000000004\" --state Faulted --output json", - "file": "7a82fd653d.json" - }, - { - "match": "or jobs list --folder-key 5a04a2b3-d4e5-4f60-8a04-000000000004 --state Faulted --output json", - "file": "7a82fd653d.json" - }, - { - "match": "or jobs get \"5b04c3d4-e5f6-4071-9b04-000000000004\" --output json", - "file": "9280daa270.json" - }, - { - "match": "or jobs get 5b04c3d4-e5f6-4071-9b04-000000000004 --output json", - "file": "9280daa270.json" - }, - { - "match": "or jobs logs \"5b04c3d4-e5f6-4071-9b04-000000000004\" --level Error --output json", - "file": "c95bb27fe0.json" - }, - { - "match": "or jobs logs 5b04c3d4-e5f6-4071-9b04-000000000004 --level Error --output json", - "file": "c95bb27fe0.json" - }, - { - "match": "or jobs logs \"5b04c3d4-e5f6-4071-9b04-000000000004\" --output json", - "file": "e00230a00f.json" - }, - { - "match": "or jobs logs 5b04c3d4-e5f6-4071-9b04-000000000004 --output json", - "file": "e00230a00f.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/uip-fixture.json new file mode 100644 index 0000000000..731e074664 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a04a2b3-d4e5-4f60-8a04-000000000004", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b04c3d4-e5f6-4071-9b04-000000000004\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraStatusUpdater\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T13:05:02.410Z\",\r\n \"EndTime\": \"2026-06-16T13:05:04.220Z\",\r\n \"CreationTime\": \"2026-06-16T13:05:01.950Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b04c3d4-e5f6-4071-9b04-000000000004", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T13:05:04.090Z\",\r\n \"Message\": \"[Transition Issue] Response status code does not indicate success: 400 (Bad Request). Transition id '31' is not valid for issue OPS-1421 in its current status.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T13:05:04.170Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Transition id '31' is not valid for issue OPS-1421 in its current status.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at TransitionIssue \\\"Transition Issue\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b04c3d4-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b04c3d4-e5f6-4071-9b04-000000000004\",\r\n \"StartTime\": \"2026-06-16T13:05:02.410Z\",\r\n \"EndTime\": \"2026-06-16T13:05:04.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c04d4e5-f6a7-4182-9c04-000000000004\",\r\n \"Info\": \"Transition Issue: Response status code does not indicate success: 400 (Bad Request). Transition id '31' is not valid for issue OPS-1421 in its current status.\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at TransitionIssue \\\"Transition Issue\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: Transition id '31' is not valid for issue OPS-1421 in its current status.\",\r\n \"CreationTime\": \"2026-06-16T13:05:01.950Z\",\r\n \"ReleaseName\": \"JiraStatusUpdater\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013304,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a04a2b3-d4e5-4f60-8a04-000000000004\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T13:05:04.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b04c3d4-e5f6-4071-9b04-000000000004\",\r\n \"Id\": 813304204\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b04c3d4-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-16T13:05:02.600Z\",\r\n \"Message\": \"JiraStatusUpdater execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T13:05:03.380Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T13:05:03.910Z\",\r\n \"Message\": \"Transition Issue: requesting transition id 31 on issue OPS-1421 (current status: In Review)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T13:05:04.090Z\",\r\n \"Message\": \"[Transition Issue] Response status code does not indicate success: 400 (Bad Request). Transition id '31' is not valid for issue OPS-1421 in its current status.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T13:05:04.170Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Transition id '31' is not valid for issue OPS-1421 in its current status.\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at TransitionIssue \\\"Transition Issue\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-16T13:05:04.220Z\",\r\n \"Message\": \"JiraStatusUpdater execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a04a2b3-d4e5-4f60-8a04-000000000004\",\r\n \"Name\": \"Issue Sync\",\r\n \"Path\": \"Issue Sync\",\r\n \"Description\": \"Production folder for Jira issue-sync processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/task.yaml index 6036af9e1a..487713ee00 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-invalid-id/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/README.md new file mode 100644 index 0000000000..6ae6f749a8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/README.md @@ -0,0 +1,49 @@ +# Transition Issue Fails - Required Field on the Transition Screen (T1) + +This scenario reproduces a `Transition Issue` failure caused by a **mandatory +field on the transition screen** not being supplied. The `Done` transition has a +screen that requires `Resolution`; the activity moves the status without it, so +Jira rejects the call with `400 (Bad Request)` and +`{"errors":{"resolution":"Field 'resolution' is required"}}`. + +## What this scenario uncovers + +**Root Cause:** `Transition Issue` moves `OPS-1530` to `Done` but supplies no +field values, and the `Done` transition screen requires `Resolution`. + +This maps to: +`references/activity-packages/jira-activities/playbooks/jira-transition-issue-failures.md` +sub-case **T1** (required field on the transition screen not supplied; use Get +Transitions / Update Issue first). + +The scope authenticates fine (so this is **not** an auth failure), the error +names a required field rather than `is not valid` (**not** the T2 invalid-ID +case), it is a field-validation rejection rather than a rule/permission denial +(**not** T3), and there is no `IssueFieldEditMetadataOperation` conversion error +(**not** T4). "Closing by hand works" because the Jira UI prompts for the +resolution. The user is framed as **off-host**, so the correct agent behavior is +to tie the rejection to the missing required field and recommend supplying it - +not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Transition Issue` to `Done` with no field values | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `jira-transition-issue-failures.md` (sub-case T1) +- Agent identified a required transition-screen field (`resolution`) not being + supplied as the cause (not an invalid transition ID, permission/validator, or + deserialization bug) and recommended supplying the field via `Get Transitions` + or `Update Issue` before the transition, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/46ac4679cc.json deleted file mode 100644 index 2b8b624956..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a05a2b3-d4e5-4f60-8a05-000000000005", - "Name": "Service Desk", - "Path": "Service Desk", - "Description": "Production folder for Jira ticket-closing processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/README.md deleted file mode 100644 index 0421633056..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Transition Issue Fails - Required Field on the Transition Screen (T1) - -This scenario reproduces a `Transition Issue` failure caused by a **mandatory -field on the transition screen** not being supplied. The `Done` transition has a -screen that requires `Resolution`; the activity moves the status without it, so -Jira rejects the call with `400 (Bad Request)` and -`{"errors":{"resolution":"Field 'resolution' is required"}}`. - -## What this scenario uncovers - -**Root Cause:** `Transition Issue` moves `OPS-1530` to `Done` but supplies no -field values, and the `Done` transition screen requires `Resolution`. - -This maps to: -`references/activity-packages/jira-activities/playbooks/jira-transition-issue-failures.md` -sub-case **T1** (required field on the transition screen not supplied; use Get -Transitions / Update Issue first). - -The scope authenticates fine (so this is **not** an auth failure), the error -names a required field rather than `is not valid` (**not** the T2 invalid-ID -case), it is a field-validation rejection rather than a rule/permission denial -(**not** T3), and there is no `IssueFieldEditMetadataOperation` conversion error -(**not** T4). "Closing by hand works" because the Jira UI prompts for the -resolution. The user is framed as **off-host**, so the correct agent behavior is -to tie the rejection to the missing required field and recommend supplying it - -not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Jira Scope` -> `Transition Issue` to `Done` with no field values | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `jira-transition-issue-failures.md` (sub-case T1) -- Agent identified a required transition-screen field (`resolution`) not being - supplied as the cause (not an invalid transition ID, permission/validator, or - deserialization bug) and recommended supplying the field via `Get Transitions` - or `Update Issue` before the transition, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/a6449ecbfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/a6449ecbfe.json deleted file mode 100644 index 48929c6cca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/a6449ecbfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T15:40:03.730Z", - "Message": "[Transition Issue] Response status code does not indicate success: 400 (Bad Request). {\"errorMessages\":[],\"errors\":{\"resolution\":\"Field 'resolution' is required\"}}" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T15:40:03.810Z", - "Message": "UiPath.Jira.Activities.JiraException: Field 'resolution' is required\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at TransitionIssue \"Transition Issue\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/b4d111927f.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/b4d111927f.json deleted file mode 100644 index dd29c56f61..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/b4d111927f.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-06-16T15:40:02.300Z", - "Message": "JiraTicketCloser execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T15:40:03.050Z", - "Message": "Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T15:40:03.520Z", - "Message": "Transition Issue: transitioning issue OPS-1530 to 'Done'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T15:40:03.730Z", - "Message": "[Transition Issue] Response status code does not indicate success: 400 (Bad Request). {\"errorMessages\":[],\"errors\":{\"resolution\":\"Field 'resolution' is required\"}}" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T15:40:03.810Z", - "Message": "UiPath.Jira.Activities.JiraException: Field 'resolution' is required\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at TransitionIssue \"Transition Issue\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Information", - "TimeStamp": "2026-06-16T15:40:03.860Z", - "Message": "JiraTicketCloser execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/bf530a488f.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/bf530a488f.json deleted file mode 100644 index 2479c8968f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/bf530a488f.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b05c3d4-e5f6-4071-9b05-000000000005", - "StartTime": "2026-06-16T15:40:02.110Z", - "EndTime": "2026-06-16T15:40:03.860Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5c05d4e5-f6a7-4182-9c05-000000000005", - "Info": "Transition Issue: Response status code does not indicate success: 400 (Bad Request). {\"errorMessages\":[],\"errors\":{\"resolution\":\"Field 'resolution' is required\"}}\r\n\r\nin Main.xaml\r\n at JiraApplicationScope \"Jira Scope\"\r\n at TransitionIssue \"Transition Issue\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Jira.Activities.JiraException: Field 'resolution' is required", - "CreationTime": "2026-06-16T15:40:01.700Z", - "ReleaseName": "JiraTicketCloser", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 8013305, - "OrganizationUnitId": 7791618, - "FolderKey": "5a05a2b3-d4e5-4f60-8a05-000000000005", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T15:40:03.860Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b05c3d4-e5f6-4071-9b05-000000000005", - "Id": 813305205 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/f758fc3355.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/f758fc3355.json deleted file mode 100644 index a7032d6dfb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/f758fc3355.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b05c3d4-e5f6-4071-9b05-000000000005", - "State": "Faulted", - "ProcessName": "JiraTicketCloser", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-16T15:40:02.110Z", - "EndTime": "2026-06-16T15:40:03.860Z", - "CreationTime": "2026-06-16T15:40:01.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/manifest.json deleted file mode 100644 index 8a86a9e5eb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs list --folder-key \"5a05a2b3-d4e5-4f60-8a05-000000000005\" --state Faulted --output json", - "file": "f758fc3355.json" - }, - { - "match": "or jobs list --folder-key 5a05a2b3-d4e5-4f60-8a05-000000000005 --state Faulted --output json", - "file": "f758fc3355.json" - }, - { - "match": "or jobs get \"5b05c3d4-e5f6-4071-9b05-000000000005\" --output json", - "file": "bf530a488f.json" - }, - { - "match": "or jobs get 5b05c3d4-e5f6-4071-9b05-000000000005 --output json", - "file": "bf530a488f.json" - }, - { - "match": "or jobs logs \"5b05c3d4-e5f6-4071-9b05-000000000005\" --level Error --output json", - "file": "a6449ecbfe.json" - }, - { - "match": "or jobs logs 5b05c3d4-e5f6-4071-9b05-000000000005 --level Error --output json", - "file": "a6449ecbfe.json" - }, - { - "match": "or jobs logs \"5b05c3d4-e5f6-4071-9b05-000000000005\" --output json", - "file": "b4d111927f.json" - }, - { - "match": "or jobs logs 5b05c3d4-e5f6-4071-9b05-000000000005 --output json", - "file": "b4d111927f.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/uip-fixture.json new file mode 100644 index 0000000000..cbc86c0c78 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a05a2b3-d4e5-4f60-8a05-000000000005", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b05c3d4-e5f6-4071-9b05-000000000005\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"JiraTicketCloser\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T15:40:02.110Z\",\r\n \"EndTime\": \"2026-06-16T15:40:03.860Z\",\r\n \"CreationTime\": \"2026-06-16T15:40:01.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b05c3d4-e5f6-4071-9b05-000000000005", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.730Z\",\r\n \"Message\": \"[Transition Issue] Response status code does not indicate success: 400 (Bad Request). {\\\"errorMessages\\\":[],\\\"errors\\\":{\\\"resolution\\\":\\\"Field 'resolution' is required\\\"}}\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.810Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Field 'resolution' is required\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at TransitionIssue \\\"Transition Issue\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b05c3d4-e5f6-4071-9b05-000000000005", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b05c3d4-e5f6-4071-9b05-000000000005\",\r\n \"StartTime\": \"2026-06-16T15:40:02.110Z\",\r\n \"EndTime\": \"2026-06-16T15:40:03.860Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5c05d4e5-f6a7-4182-9c05-000000000005\",\r\n \"Info\": \"Transition Issue: Response status code does not indicate success: 400 (Bad Request). {\\\"errorMessages\\\":[],\\\"errors\\\":{\\\"resolution\\\":\\\"Field 'resolution' is required\\\"}}\\r\\n\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at TransitionIssue \\\"Transition Issue\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Jira.Activities.JiraException: Field 'resolution' is required\",\r\n \"CreationTime\": \"2026-06-16T15:40:01.700Z\",\r\n \"ReleaseName\": \"JiraTicketCloser\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 8013305,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a05a2b3-d4e5-4f60-8a05-000000000005\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T15:40:03.860Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b05c3d4-e5f6-4071-9b05-000000000005\",\r\n \"Id\": 813305205\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b05c3d4-e5f6-4071-9b05-000000000005", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-16T15:40:02.300Z\",\r\n \"Message\": \"JiraTicketCloser execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.050Z\",\r\n \"Message\": \"Jira Scope: session opened to https://acme.atlassian.net (Authentication Type: Api Token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.520Z\",\r\n \"Message\": \"Transition Issue: transitioning issue OPS-1530 to 'Done'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.730Z\",\r\n \"Message\": \"[Transition Issue] Response status code does not indicate success: 400 (Bad Request). {\\\"errorMessages\\\":[],\\\"errors\\\":{\\\"resolution\\\":\\\"Field 'resolution' is required\\\"}}\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.810Z\",\r\n \"Message\": \"UiPath.Jira.Activities.JiraException: Field 'resolution' is required\\r\\nin Main.xaml\\r\\n at JiraApplicationScope \\\"Jira Scope\\\"\\r\\n at TransitionIssue \\\"Transition Issue\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-06-16T15:40:03.860Z\",\r\n \"Message\": \"JiraTicketCloser execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a05a2b3-d4e5-4f60-8a05-000000000005\",\r\n \"Name\": \"Service Desk\",\r\n \"Path\": \"Service Desk\",\r\n \"Description\": \"Production folder for Jira ticket-closing processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/task.yaml index ca178408a9..e4408fc79f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/jira-transition-required-field/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/README.md new file mode 100644 index 0000000000..bd6af5d9ae --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/README.md @@ -0,0 +1,68 @@ +# Delete Outlook Mail Failure - Collection Modified Inside the Loop (Branch 2) + +This scenario reproduces a runtime `Delete Outlook Mail Message` failure caused +by deleting **inside a `For Each`** that iterates the live `List` +returned by `Get Outlook Mail Messages`. Deleting the current item mutates the +collection being enumerated, so the next iteration raises +`System.InvalidOperationException: Collection was modified; enumeration +operation may not execute.` + +## What this scenario uncovers + +**Root Cause:** In `Main.xaml`, `Get Outlook Mail Messages` reads the Inbox into +the `List` variable `mailList`. A `For Each` (`ForEach`) +iterates `mailList`, and `Delete Outlook Mail Message` deletes the current `item` +**directly inside that loop**. The first delete succeeds; on the next iteration +the `For Each` enumerator detects the collection changed and throws +`Collection was modified; enumeration operation may not execute.` + +It is a **workflow-structure bug**, not an Outlook / COM / mailbox problem. + +This maps to: +`references/activity-packages/mail-activities/playbooks/delete-outlook-mail-failures.md` +(specifically **Branch 2 — Collection modified inside the loop**). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `Delete Outlook Mail Message` nested inside a `For Each` over `mailList` | + +The smoking gun is in `or-jobs-logs-e6f7a8b9-output-json.json`: an Info that Get +retrieved 37 messages into `mailList`, an Info that the `For Each` began +iterating, an Info that message 1 of 37 was deleted, then the +`Collection was modified` Error on the next iteration — the delete-inside-loop +mutated the collection mid-enumeration. + +> **Note on fixtures.** Fixtures here were authored from the documented playbook +> Branch 2 signature rather than captured from a real `.local/investigations/` +> session. + +## Sibling-branch comparison (delete-outlook-mail-failures playbook) + +| Branch | Signature | This scenario? | Why not | +|---|---|---|---| +| 1 — Stale message reference | `The operation failed. An object could not be found.` | No | Error is `Collection was modified`; the first delete succeeded | +| **2 — Collection modified in loop** | `Collection was modified; enumeration operation may not execute.` inside a For Each | **Yes** | Delete runs inside For Each over the live `mailList` | +| 3 — New Outlook (COM API removed) | Broke completely after an Office/system update; COM-bind failure | No | Messages fetched and one deleted — COM surface works | +| 4 — COM session blocked / privilege | `The operation has timed out` / freeze | No | Immediate deterministic `InvalidOperationException`, no timeout | +| 5 — Mailbox permission / access denied | Access-denied on a (shared) mailbox | No | Account read the Inbox and deleted the first message | + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `delete-outlook-mail-failures.md` Branch 2 (collection modified + inside the loop). +- Agent attributed the fault to deleting inside the `For Each` over the live + `mailList`, and recommended iterating a `.ToList()` copy / backwards by index, + or Move-then-delete in a second pass — **NOT** merely wrapping the loop in a + `Try Catch` (the enumeration is already broken). + +This scenario is workflow-fixable from the project source and the Orchestrator +evidence alone; no host access is required. The judge **scores the conclusion, +not the trajectory** — any investigation path that reaches the Branch 2 root +cause and the safe-iteration fix passes. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/2befe51465.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/2befe51465.json deleted file mode 100644 index 70d35e080f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/2befe51465.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T13:30:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T13:30:02.640Z", - "Message": "[Get Outlook Mail Messages] Retrieved 37 messages from folder Inbox into mailList" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T13:30:02.910Z", - "Message": "[For Each] Begin iterating mailList (37 items)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T13:30:03.880Z", - "Message": "[Delete Outlook Mail Message] Deleted message 1 of 37 (Subject: 'Weekly status digest')" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T13:30:04.402Z", - "Message": "[Delete Outlook Mail Message] Collection was modified; enumeration operation may not execute." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T13:30:04.451Z", - "Message": "Collection was modified; enumeration operation may not execute.\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at ForEach \"For Each\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T13:30:04.465Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/927e7f42a9.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/927e7f42a9.json deleted file mode 100644 index 266201b60f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/927e7f42a9.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10", - "State": "Faulted", - "ProcessName": "BulkMailPurger", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T13:30:00.140Z", - "EndTime": "2026-06-04T13:30:04.470Z", - "CreationTime": "2026-06-04T13:29:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/README.md deleted file mode 100644 index b8d5a959eb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# Delete Outlook Mail Failure - Collection Modified Inside the Loop (Branch 2) - -This scenario reproduces a runtime `Delete Outlook Mail Message` failure caused -by deleting **inside a `For Each`** that iterates the live `List` -returned by `Get Outlook Mail Messages`. Deleting the current item mutates the -collection being enumerated, so the next iteration raises -`System.InvalidOperationException: Collection was modified; enumeration -operation may not execute.` - -## What this scenario uncovers - -**Root Cause:** In `Main.xaml`, `Get Outlook Mail Messages` reads the Inbox into -the `List` variable `mailList`. A `For Each` (`ForEach`) -iterates `mailList`, and `Delete Outlook Mail Message` deletes the current `item` -**directly inside that loop**. The first delete succeeds; on the next iteration -the `For Each` enumerator detects the collection changed and throws -`Collection was modified; enumeration operation may not execute.` - -It is a **workflow-structure bug**, not an Outlook / COM / mailbox problem. - -This maps to: -`references/activity-packages/mail-activities/playbooks/delete-outlook-mail-failures.md` -(specifically **Branch 2 — Collection modified inside the loop**). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `Delete Outlook Mail Message` nested inside a `For Each` over `mailList` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table (quoted + unquoted arg forms) | - -The smoking gun is in `or-jobs-logs-e6f7a8b9-output-json.json`: an Info that Get -retrieved 37 messages into `mailList`, an Info that the `For Each` began -iterating, an Info that message 1 of 37 was deleted, then the -`Collection was modified` Error on the next iteration — the delete-inside-loop -mutated the collection mid-enumeration. - -> **Note on fixtures.** Fixtures here were authored from the documented playbook -> Branch 2 signature rather than captured from a real `.local/investigations/` -> session. - -## Sibling-branch comparison (delete-outlook-mail-failures playbook) - -| Branch | Signature | This scenario? | Why not | -|---|---|---|---| -| 1 — Stale message reference | `The operation failed. An object could not be found.` | No | Error is `Collection was modified`; the first delete succeeded | -| **2 — Collection modified in loop** | `Collection was modified; enumeration operation may not execute.` inside a For Each | **Yes** | Delete runs inside For Each over the live `mailList` | -| 3 — New Outlook (COM API removed) | Broke completely after an Office/system update; COM-bind failure | No | Messages fetched and one deleted — COM surface works | -| 4 — COM session blocked / privilege | `The operation has timed out` / freeze | No | Immediate deterministic `InvalidOperationException`, no timeout | -| 5 — Mailbox permission / access denied | Access-denied on a (shared) mailbox | No | Account read the Inbox and deleted the first message | - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `delete-outlook-mail-failures.md` Branch 2 (collection modified - inside the loop). -- Agent attributed the fault to deleting inside the `For Each` over the live - `mailList`, and recommended iterating a `.ToList()` copy / backwards by index, - or Move-then-delete in a second pass — **NOT** merely wrapping the loop in a - `Try Catch` (the enumeration is already broken). - -This scenario is workflow-fixable from the project source and the Orchestrator -evidence alone; no host access is required. The judge **scores the conclusion, -not the trajectory** — any investigation path that reaches the Branch 2 root -cause and the safe-iteration fix passes. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/a0139f5005.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/a0139f5005.json deleted file mode 100644 index df09faac92..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/a0139f5005.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T13:30:04.402Z", - "Message": "[Delete Outlook Mail Message] Collection was modified; enumeration operation may not execute." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T13:30:04.451Z", - "Message": "Collection was modified; enumeration operation may not execute.\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at ForEach \"For Each\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/a961df0b22.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/a961df0b22.json deleted file mode 100644 index b01a137224..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/a961df0b22.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the mailbox-maintenance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/cc6dd9e0a0.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/cc6dd9e0a0.json deleted file mode 100644 index 628518b421..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/cc6dd9e0a0.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10", - "StartTime": "2026-06-04T13:30:00.140Z", - "EndTime": "2026-06-04T13:30:04.470Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "a8b9c0d1-2345-4e6f-8a9b-0c1d2e3f4a5b", - "Info": "Delete Outlook Mail Message: Collection was modified; enumeration operation may not execute.\r\n\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at ForEach \"For Each\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidOperationException: Collection was modified; enumeration operation may not execute.", - "CreationTime": "2026-06-04T13:29:59.910Z", - "ReleaseName": "BulkMailPurger", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012044, - "OrganizationUnitId": 7791618, - "FolderKey": "d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T13:30:04.470Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10", - "Id": 691204503 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/manifest.json deleted file mode 100644 index df7ee66e00..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Delete Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "a961df0b22.json" - }, - { - "match": "or jobs list --folder-key \"d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f\" --state Faulted --output json", - "file": "927e7f42a9.json" - }, - { - "match": "or jobs list --folder-key d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f --state Faulted --output json", - "file": "927e7f42a9.json" - }, - { - "match": "or jobs get \"e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10\" --output json", - "file": "cc6dd9e0a0.json" - }, - { - "match": "or jobs get e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10 --output json", - "file": "cc6dd9e0a0.json" - }, - { - "match": "or jobs logs \"e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10\" --level Error --output json", - "file": "a0139f5005.json" - }, - { - "match": "or jobs logs e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10 --level Error --output json", - "file": "a0139f5005.json" - }, - { - "match": "or jobs logs \"e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10\" --output json", - "file": "2befe51465.json" - }, - { - "match": "or jobs logs e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10 --output json", - "file": "2befe51465.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/uip-fixture.json new file mode 100644 index 0000000000..4dff64377d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BulkMailPurger\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T13:30:00.140Z\",\r\n \"EndTime\": \"2026-06-04T13:30:04.470Z\",\r\n \"CreationTime\": \"2026-06-04T13:29:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T13:30:04.402Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] Collection was modified; enumeration operation may not execute.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T13:30:04.451Z\",\r\n \"Message\": \"Collection was modified; enumeration operation may not execute.\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at ForEach \\\"For Each\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10\",\r\n \"StartTime\": \"2026-06-04T13:30:00.140Z\",\r\n \"EndTime\": \"2026-06-04T13:30:04.470Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"a8b9c0d1-2345-4e6f-8a9b-0c1d2e3f4a5b\",\r\n \"Info\": \"Delete Outlook Mail Message: Collection was modified; enumeration operation may not execute.\\r\\n\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at ForEach \\\"For Each\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidOperationException: Collection was modified; enumeration operation may not execute.\",\r\n \"CreationTime\": \"2026-06-04T13:29:59.910Z\",\r\n \"ReleaseName\": \"BulkMailPurger\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012044,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T13:30:04.470Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10\",\r\n \"Id\": 691204503\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e6f7a8b9-0123-4d4e-9f5a-6b7c8d9e0f10", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T13:30:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T13:30:02.640Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Retrieved 37 messages from folder Inbox into mailList\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T13:30:02.910Z\",\r\n \"Message\": \"[For Each] Begin iterating mailList (37 items)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T13:30:03.880Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] Deleted message 1 of 37 (Subject: 'Weekly status digest')\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T13:30:04.402Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] Collection was modified; enumeration operation may not execute.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T13:30:04.451Z\",\r\n \"Message\": \"Collection was modified; enumeration operation may not execute.\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at ForEach \\\"For Each\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T13:30:04.465Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5e6f7a8-9012-4c3d-8e4f-5a6b7c8d9e0f\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the mailbox-maintenance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/task.yaml index 6182cab82e..4a5d54c088 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-collection-modified/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/README.md new file mode 100644 index 0000000000..ff0549395a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/README.md @@ -0,0 +1,68 @@ +# Delete Outlook Mail Message Failure - Shared-Mailbox Permission Denied (Branch 5) + +This scenario reproduces a runtime `Delete Outlook Mail Message` +(`DeleteOutlookMailMessage`, `UiPath.Mail.Outlook.Activities`) failure caused +by a **mailbox-permission / access-denied** error on a **shared mailbox**. +The shared mailbox `ap-shared@contoso.com` opens and the target message is +found, but the delete is denied because the Robot account has **Send-As** but +**not Full Access (delete)** rights. The COM layer surfaces it as +`You do not have sufficient permission to perform this operation on this +object.` / `Access is denied. (Exception from HRESULT: 0x80070005 +(E_ACCESSDENIED))`. + +## What this scenario uncovers + +**Root Cause:** The `Delete Outlook Mail Message` activity in `Main.xaml` +targets a message in the shared mailbox `ap-shared@contoso.com` with that +address set in `Account`. The read (`Get Outlook Mail Messages`) succeeds and +the message resolves, but the delete is denied: the Robot account +`UIPATH\ROBOTUSER1` has Send-As but not Full Access on the shared mailbox, so +it cannot delete. Send-As does not grant delete rights — only Full Access +does. + +This maps to: +`references/activity-packages/mail-activities/playbooks/delete-outlook-mail-failures.md` +Branch 5 (Mailbox permission / access denied). + +The user is framed as **off-host and not an Exchange admin**, so the correct +agent behavior is to hand an IT / Exchange check list (keep `Account` as the +shared address; request Full Access for the Robot's AD account from the +Exchange admin) rather than fabricate an Exchange change it cannot make. + +## Sibling-branch comparison (delete-outlook-mail-failures) + +| Branch | Signature | Why NOT this scenario | +|---|---|---| +| 1 — Stale message reference | `An object could not be found` after a fetch/delete gap | The target message WAS resolved/found right before the delete | +| 2 — Collection modified in loop | `Collection was modified; enumeration operation may not execute` inside a `For Each` | Single delete after Get; no loop mutating the live collection | +| 3 — New Outlook | Worked, then broke completely after an update; COM-bind failure | COM call reaches Outlook, opens the mailbox, and is rejected with an access error | +| 4 — COM session blocked / privilege | `The operation has timed out` / freeze, no clean exception | Fast, clean access-denied exception — not a hang | +| **5 — Mailbox permission / access denied** | **`insufficient permission` / `Access is denied (0x80070005 E_ACCESSDENIED)` on a shared mailbox** | **This scenario: Send-As but not Full Access on `ap-shared@contoso.com`** | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: `Get Outlook Mail Messages` then `Delete Outlook Mail Message`, both with `Account="ap-shared@contoso.com"` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature (Branch 5) rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This scenario **scores the conclusion, not the trajectory**. The only graded +outcomes are: + +- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). +- Agent matched `delete-outlook-mail-failures.md` Branch 5 and reached the + same conclusion as `RESOLUTION.md`: the delete was denied because the Robot + account lacks Full Access (delete) on the shared mailbox (Send-As is not + enough), the folder/`Account` are fine and the message was found, and the + fix is to keep `Account` as the shared address and have the Exchange admin + grant Full Access — handed as an IT check list, with no fabricated Exchange + or host action (`llm_judge`). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/2a09500c11.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/2a09500c11.json deleted file mode 100644 index c2a3ac0d5a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/2a09500c11.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132", - "StartTime": "2026-06-04T15:15:00.090Z", - "EndTime": "2026-06-04T15:15:03.140Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "4b365cf1-3661-4e3d-b88b-b848de654454", - "Info": "Delete Outlook Mail Message: You do not have sufficient permission to perform this operation on this object.\r\n\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))", - "CreationTime": "2026-06-04T15:14:59.870Z", - "ReleaseName": "SharedInboxArchiver", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020114, - "OrganizationUnitId": 7791618, - "FolderKey": "f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T15:15:03.140Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132", - "Id": 682319114 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/39f21518dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/39f21518dc.json deleted file mode 100644 index ae0de5ea8e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/39f21518dc.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132", - "State": "Faulted", - "ProcessName": "SharedInboxArchiver", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T15:15:00.090Z", - "EndTime": "2026-06-04T15:15:03.140Z", - "CreationTime": "2026-06-04T15:14:59.870Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/818023044c.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/818023044c.json deleted file mode 100644 index 3b7709899e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/818023044c.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T15:15:03.041Z", - "Message": "[Delete Outlook Mail Message] delete denied on shared mailbox 'ap-shared@contoso.com'; the account can send-as but lacks Full Access (delete) rights" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T15:15:03.072Z", - "Message": "[Delete Outlook Mail Message] You do not have sufficient permission to perform this operation on this object." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T15:15:03.108Z", - "Message": "Delete Outlook Mail Message: You do not have sufficient permission to perform this operation on this object.\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\nSystem.Runtime.InteropServices.COMException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/README.md deleted file mode 100644 index c98e05f57c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# Delete Outlook Mail Message Failure - Shared-Mailbox Permission Denied (Branch 5) - -This scenario reproduces a runtime `Delete Outlook Mail Message` -(`DeleteOutlookMailMessage`, `UiPath.Mail.Outlook.Activities`) failure caused -by a **mailbox-permission / access-denied** error on a **shared mailbox**. -The shared mailbox `ap-shared@contoso.com` opens and the target message is -found, but the delete is denied because the Robot account has **Send-As** but -**not Full Access (delete)** rights. The COM layer surfaces it as -`You do not have sufficient permission to perform this operation on this -object.` / `Access is denied. (Exception from HRESULT: 0x80070005 -(E_ACCESSDENIED))`. - -## What this scenario uncovers - -**Root Cause:** The `Delete Outlook Mail Message` activity in `Main.xaml` -targets a message in the shared mailbox `ap-shared@contoso.com` with that -address set in `Account`. The read (`Get Outlook Mail Messages`) succeeds and -the message resolves, but the delete is denied: the Robot account -`UIPATH\ROBOTUSER1` has Send-As but not Full Access on the shared mailbox, so -it cannot delete. Send-As does not grant delete rights — only Full Access -does. - -This maps to: -`references/activity-packages/mail-activities/playbooks/delete-outlook-mail-failures.md` -Branch 5 (Mailbox permission / access denied). - -The user is framed as **off-host and not an Exchange admin**, so the correct -agent behavior is to hand an IT / Exchange check list (keep `Account` as the -shared address; request Full Access for the Robot's AD account from the -Exchange admin) rather than fabricate an Exchange change it cannot make. - -## Sibling-branch comparison (delete-outlook-mail-failures) - -| Branch | Signature | Why NOT this scenario | -|---|---|---| -| 1 — Stale message reference | `An object could not be found` after a fetch/delete gap | The target message WAS resolved/found right before the delete | -| 2 — Collection modified in loop | `Collection was modified; enumeration operation may not execute` inside a `For Each` | Single delete after Get; no loop mutating the live collection | -| 3 — New Outlook | Worked, then broke completely after an update; COM-bind failure | COM call reaches Outlook, opens the mailbox, and is rejected with an access error | -| 4 — COM session blocked / privilege | `The operation has timed out` / freeze, no clean exception | Fast, clean access-denied exception — not a hang | -| **5 — Mailbox permission / access denied** | **`insufficient permission` / `Access is denied (0x80070005 E_ACCESSDENIED)` on a shared mailbox** | **This scenario: Send-As but not Full Access on `ap-shared@contoso.com`** | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: `Get Outlook Mail Messages` then `Delete Outlook Mail Message`, both with `Account="ap-shared@contoso.com"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the Branch 5 playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature (Branch 5) rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This scenario **scores the conclusion, not the trajectory**. The only graded -outcomes are: - -- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). -- Agent matched `delete-outlook-mail-failures.md` Branch 5 and reached the - same conclusion as `RESOLUTION.md`: the delete was denied because the Robot - account lacks Full Access (delete) on the shared mailbox (Send-As is not - enough), the folder/`Account` are fine and the message was found, and the - fix is to keep `Account` as the shared address and have the Exchange admin - grant Full Access — handed as an IT check list, with no fabricated Exchange - or host action (`llm_judge`). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/a5f14e57b4.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/a5f14e57b4.json deleted file mode 100644 index 39e74b9f59..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/a5f14e57b4.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-5a6b-4789-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/a84a8d8190.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/a84a8d8190.json deleted file mode 100644 index 42050ab4a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/a84a8d8190.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T15:15:00.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T15:15:01.220Z", - "Message": "[Get Outlook Mail Messages] Opened shared mailbox 'ap-shared@contoso.com' (Inbox)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T15:15:02.004Z", - "Message": "[Get Outlook Mail Messages] Retrieved 7 message(s) from 'ap-shared@contoso.com\\Inbox'" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T15:15:02.461Z", - "Message": "[Delete Outlook Mail Message] Target message resolved in 'ap-shared@contoso.com\\Inbox' (Subject: 'Invoice batch 0604'); attempting delete" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T15:15:03.041Z", - "Message": "[Delete Outlook Mail Message] delete denied on shared mailbox 'ap-shared@contoso.com'; the account can send-as but lacks Full Access (delete) rights" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T15:15:03.072Z", - "Message": "[Delete Outlook Mail Message] You do not have sufficient permission to perform this operation on this object." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T15:15:03.108Z", - "Message": "Delete Outlook Mail Message: You do not have sufficient permission to perform this operation on this object.\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\nSystem.Runtime.InteropServices.COMException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T15:15:03.135Z", - "Message": "Execution ended in: 00:00:03" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/manifest.json deleted file mode 100644 index cab32d688d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Delete Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "a5f14e57b4.json" - }, - { - "match": "or jobs list --folder-key \"f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021\" --state Faulted --output json", - "file": "39f21518dc.json" - }, - { - "match": "or jobs list --folder-key f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021 --state Faulted --output json", - "file": "39f21518dc.json" - }, - { - "match": "or jobs get \"a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132\" --output json", - "file": "2a09500c11.json" - }, - { - "match": "or jobs get a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132 --output json", - "file": "2a09500c11.json" - }, - { - "match": "or jobs logs \"a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132\" --level Error --output json", - "file": "818023044c.json" - }, - { - "match": "or jobs logs a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132 --level Error --output json", - "file": "818023044c.json" - }, - { - "match": "or jobs logs \"a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132\" --output json", - "file": "a84a8d8190.json" - }, - { - "match": "or jobs logs a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132 --output json", - "file": "a84a8d8190.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/uip-fixture.json new file mode 100644 index 0000000000..fe99a06d8a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SharedInboxArchiver\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T15:15:00.090Z\",\r\n \"EndTime\": \"2026-06-04T15:15:03.140Z\",\r\n \"CreationTime\": \"2026-06-04T15:14:59.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.041Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] delete denied on shared mailbox 'ap-shared@contoso.com'; the account can send-as but lacks Full Access (delete) rights\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.072Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] You do not have sufficient permission to perform this operation on this object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.108Z\",\r\n \"Message\": \"Delete Outlook Mail Message: You do not have sufficient permission to perform this operation on this object.\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\nSystem.Runtime.InteropServices.COMException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132\",\r\n \"StartTime\": \"2026-06-04T15:15:00.090Z\",\r\n \"EndTime\": \"2026-06-04T15:15:03.140Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"4b365cf1-3661-4e3d-b88b-b848de654454\",\r\n \"Info\": \"Delete Outlook Mail Message: You do not have sufficient permission to perform this operation on this object.\\r\\n\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))\",\r\n \"CreationTime\": \"2026-06-04T15:14:59.870Z\",\r\n \"ReleaseName\": \"SharedInboxArchiver\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020114,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T15:15:03.140Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132\",\r\n \"Id\": 682319114\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a8b9c0d1-2345-4f6a-9b7c-8d9e0f102132", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T15:15:00.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T15:15:01.220Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Opened shared mailbox 'ap-shared@contoso.com' (Inbox)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T15:15:02.004Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Retrieved 7 message(s) from 'ap-shared@contoso.com\\\\Inbox'\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T15:15:02.461Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] Target message resolved in 'ap-shared@contoso.com\\\\Inbox' (Subject: 'Invoice batch 0604'); attempting delete\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.041Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] delete denied on shared mailbox 'ap-shared@contoso.com'; the account can send-as but lacks Full Access (delete) rights\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.072Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] You do not have sufficient permission to perform this operation on this object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.108Z\",\r\n \"Message\": \"Delete Outlook Mail Message: You do not have sufficient permission to perform this operation on this object.\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\nSystem.Runtime.InteropServices.COMException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T15:15:03.135Z\",\r\n \"Message\": \"Execution ended in: 00:00:03\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7a8b9c0-1234-4e5f-8a6b-7c8d9e0f1021\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-5a6b-4789-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/task.yaml index d99e201275..ecb5ae1087 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-permission-denied/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/README.md new file mode 100644 index 0000000000..55ecd50b58 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/README.md @@ -0,0 +1,74 @@ +# Delete Outlook Mail Message Failure - Stale Message Reference (Branch 1) + +This scenario reproduces a runtime `Delete Outlook Mail Message` +(`DeleteOutlookMailMessage`, `UiPath.Mail.Outlook.Activities`) failure +caused by a **stale message reference**: the `MailMessage` retrieved by an +upstream `Get Outlook Mail Messages` is moved or deleted out of its folder +before the delete fires, so the activity can no longer find it and faults +with `The operation failed. An object could not be found.` + +It validates **Branch 1** of +`references/activity-packages/mail-activities/playbooks/delete-outlook-mail-failures.md`. + +## What this scenario uncovers + +**Root Cause:** In `Main.xaml`, `Get Outlook Mail Messages` reads `Inbox` +into `mailList`, the oldest item is selected as `mailToDelete`, the workflow +then runs a long per-item `For Each` processing loop (with a `Delay`), and +only **after** the loop does `Delete Outlook Mail Message` try to delete +`mailToDelete`. The item was valid at fetch time (11:00:03) but had been +moved/deleted by the time the delete ran (~11:00:08). The visible +**fetch -> delete gap** plus the job-log line "target message ... not found +in 'Inbox'; it appears to have been moved or deleted since retrieval at +11:00:03" are the smoking gun. + +**Fix (per the playbook):** fetch-then-delete promptly (do not hold a +`MailMessage` across long work), and wrap the delete in a `Try Catch` that +tolerates an already-moved / already-deleted item; re-confirm under folder +contention. **Adding a Delay before the delete is the wrong answer** -- it +widens the gap and makes the failure more likely. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: Get -> select -> long `For Each` loop -> Delete (after the loop) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +The job key / TraceId is `c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e` (short key +`c4d5e6f7`); the `RPA Production` folder key is +`b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d`. The failing job is Unattended on +`MOCK-HOST` under `UIPATH\ROBOTUSER1`, faulting ~8s after start. + +## Sibling-branch comparison (why this is Branch 1, not 2-5) + +| Branch | Signature | Distinguishing evidence here | +|--------|-----------|------------------------------| +| **B1 (this)** Stale message reference | `An object could not be found` + fetch->delete gap | Get at 11:00:03, Delete at 11:00:08 after a loop; "moved or deleted since retrieval" log line | +| B2 Collection modified in loop | `Collection was modified; enumeration operation may not execute.` | Delete runs **after** the `For Each`, not inside it; no enumeration error | +| B3 New Outlook (COM API removed) | Complete post-update break, COM-bind failure | Activity bound to Outlook fine; failure is item-specific, not total | +| B4 COM session blocked / privilege | Timeout / freeze, no clean exception | Clean, specific exception; no timeout or hang | +| B5 Mailbox permission / access | Access-denied / delete-denied (often shared mailbox) | Error is **"not found"**, not "access denied"; no shared-mailbox signal | + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `delete-outlook-mail-failures.md` / Branch 1 (stale message + reference) and read the playbook under + `references/.../mail-activities/playbooks/`. +- Agent attributed the fault to the item being moved/deleted between Get and + Delete (the gap/concurrency window) and recommended fetch-then-delete + promptly + a Try Catch tolerating the already-gone item -- **not** adding + a Delay, and **not** a sibling branch (collection-modified, New Outlook, + COM-blocked, or access-denied). + +This scenario **scores the conclusion, not the trajectory**: the LLM judge +grades only the agent's final response and tool calls against +`RESOLUTION.md`, not any intermediate `.local/investigations/` state. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/40b2fac2da.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/40b2fac2da.json deleted file mode 100644 index e5507493e2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/40b2fac2da.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:03.041Z", - "Message": "[Get Outlook Mail Messages] retrieved 24 messages from 'Inbox' at 11:00:03" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:03.120Z", - "Message": "[Pick oldest message] target for deletion selected: EntryID 0000004A8F2C1B9D7E3F4A5B6C7D8E9F00112233; Subject 'Daily export - region EMEA'; received 2026-06-04T06:12:00Z; current location 'Inbox'" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:03.260Z", - "Message": "[Process each message] Processing message: Daily export - region EMEA" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:04.910Z", - "Message": "[Process each message] Processing message: Daily export - region APAC" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:06.380Z", - "Message": "[Process each message] Processing message: Weekly reconciliation summary" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:07.840Z", - "Message": "[Process each message] processed 24 of 24 messages in 00:00:04.7 (downstream API + attachment save per item)" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-04T11:00:08.060Z", - "Message": "[Delete Outlook Mail Message] target message (EntryID 0000004A8F2C1B9D7E3F4A5B6C7D8E9F00112233) not found in 'Inbox'; it appears to have been moved or deleted since retrieval at 11:00:03" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T11:00:08.150Z", - "Message": "[Delete Outlook Mail Message] The operation failed. An object could not be found." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T11:00:08.198Z", - "Message": "The operation failed. An object could not be found.\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T11:00:08.210Z", - "Message": "Execution ended in: 00:00:08" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/5029698be8.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/5029698be8.json deleted file mode 100644 index 1417070281..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/5029698be8.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e", - "StartTime": "2026-06-04T11:00:00.140Z", - "EndTime": "2026-06-04T11:00:08.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "d5e6f708-1923-4a3b-8c4d-5e6f7a8b9c0d", - "Info": "Delete Outlook Mail Message: The operation failed. An object could not be found.\r\n\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The operation failed. An object could not be found. ---> System.Runtime.InteropServices.COMException: The operation failed. An object could not be found. (Exception from HRESULT: 0x8004010F)", - "CreationTime": "2026-06-04T10:59:59.910Z", - "ReleaseName": "InboxCleanupBot", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7042918, - "OrganizationUnitId": 7791618, - "FolderKey": "b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T11:00:08.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e", - "Id": 793428713 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/7de349b5cb.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/7de349b5cb.json deleted file mode 100644 index 9347fcfd0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/7de349b5cb.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e", - "State": "Faulted", - "ProcessName": "InboxCleanupBot", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T11:00:00.140Z", - "EndTime": "2026-06-04T11:00:08.220Z", - "CreationTime": "2026-06-04T10:59:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/README.md deleted file mode 100644 index 661d7a9e18..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/README.md +++ /dev/null @@ -1,73 +0,0 @@ -# Delete Outlook Mail Message Failure - Stale Message Reference (Branch 1) - -This scenario reproduces a runtime `Delete Outlook Mail Message` -(`DeleteOutlookMailMessage`, `UiPath.Mail.Outlook.Activities`) failure -caused by a **stale message reference**: the `MailMessage` retrieved by an -upstream `Get Outlook Mail Messages` is moved or deleted out of its folder -before the delete fires, so the activity can no longer find it and faults -with `The operation failed. An object could not be found.` - -It validates **Branch 1** of -`references/activity-packages/mail-activities/playbooks/delete-outlook-mail-failures.md`. - -## What this scenario uncovers - -**Root Cause:** In `Main.xaml`, `Get Outlook Mail Messages` reads `Inbox` -into `mailList`, the oldest item is selected as `mailToDelete`, the workflow -then runs a long per-item `For Each` processing loop (with a `Delay`), and -only **after** the loop does `Delete Outlook Mail Message` try to delete -`mailToDelete`. The item was valid at fetch time (11:00:03) but had been -moved/deleted by the time the delete ran (~11:00:08). The visible -**fetch -> delete gap** plus the job-log line "target message ... not found -in 'Inbox'; it appears to have been moved or deleted since retrieval at -11:00:03" are the smoking gun. - -**Fix (per the playbook):** fetch-then-delete promptly (do not hold a -`MailMessage` across long work), and wrap the delete in a `Try Catch` that -tolerates an already-moved / already-deleted item; re-confirm under folder -contention. **Adding a Delay before the delete is the wrong answer** -- it -widens the gap and makes the failure more likely. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: Get -> select -> long `For Each` loop -> Delete (after the loop) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -The job key / TraceId is `c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e` (short key -`c4d5e6f7`); the `RPA Production` folder key is -`b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d`. The failing job is Unattended on -`MOCK-HOST` under `UIPATH\ROBOTUSER1`, faulting ~8s after start. - -## Sibling-branch comparison (why this is Branch 1, not 2-5) - -| Branch | Signature | Distinguishing evidence here | -|--------|-----------|------------------------------| -| **B1 (this)** Stale message reference | `An object could not be found` + fetch->delete gap | Get at 11:00:03, Delete at 11:00:08 after a loop; "moved or deleted since retrieval" log line | -| B2 Collection modified in loop | `Collection was modified; enumeration operation may not execute.` | Delete runs **after** the `For Each`, not inside it; no enumeration error | -| B3 New Outlook (COM API removed) | Complete post-update break, COM-bind failure | Activity bound to Outlook fine; failure is item-specific, not total | -| B4 COM session blocked / privilege | Timeout / freeze, no clean exception | Clean, specific exception; no timeout or hang | -| B5 Mailbox permission / access | Access-denied / delete-denied (often shared mailbox) | Error is **"not found"**, not "access denied"; no shared-mailbox signal | - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `delete-outlook-mail-failures.md` / Branch 1 (stale message - reference) and read the playbook under - `references/.../mail-activities/playbooks/`. -- Agent attributed the fault to the item being moved/deleted between Get and - Delete (the gap/concurrency window) and recommended fetch-then-delete - promptly + a Try Catch tolerating the already-gone item -- **not** adding - a Delay, and **not** a sibling branch (collection-modified, New Outlook, - COM-blocked, or access-denied). - -This scenario **scores the conclusion, not the trajectory**: the LLM judge -grades only the agent's final response and tool calls against -`RESOLUTION.md`, not any intermediate `.local/investigations/` state. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/cb9c67f5bc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/cb9c67f5bc.json deleted file mode 100644 index b1e1818ef1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/cb9c67f5bc.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the mailbox automation portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/e7317278fc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/e7317278fc.json deleted file mode 100644 index c77a199930..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/e7317278fc.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T11:00:08.150Z", - "Message": "[Delete Outlook Mail Message] The operation failed. An object could not be found." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T11:00:08.198Z", - "Message": "The operation failed. An object could not be found.\r\nin Main.xaml\r\n at DeleteOutlookMailMessage \"Delete Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/manifest.json deleted file mode 100644 index 02dcd9977a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Delete Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "cb9c67f5bc.json" - }, - { - "match": "or jobs list --folder-key \"b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d\" --state Faulted --output json", - "file": "7de349b5cb.json" - }, - { - "match": "or jobs list --folder-key b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d --state Faulted --output json", - "file": "7de349b5cb.json" - }, - { - "match": "or jobs get \"c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e\" --output json", - "file": "5029698be8.json" - }, - { - "match": "or jobs get c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e --output json", - "file": "5029698be8.json" - }, - { - "match": "or jobs logs \"c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e\" --level Error --output json", - "file": "e7317278fc.json" - }, - { - "match": "or jobs logs c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e --level Error --output json", - "file": "e7317278fc.json" - }, - { - "match": "or jobs logs \"c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e\" --output json", - "file": "40b2fac2da.json" - }, - { - "match": "or jobs logs c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e --output json", - "file": "40b2fac2da.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/uip-fixture.json new file mode 100644 index 0000000000..b5449b7f66 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxCleanupBot\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T11:00:00.140Z\",\r\n \"EndTime\": \"2026-06-04T11:00:08.220Z\",\r\n \"CreationTime\": \"2026-06-04T10:59:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T11:00:08.150Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] The operation failed. An object could not be found.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T11:00:08.198Z\",\r\n \"Message\": \"The operation failed. An object could not be found.\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"StartTime\": \"2026-06-04T11:00:00.140Z\",\r\n \"EndTime\": \"2026-06-04T11:00:08.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"d5e6f708-1923-4a3b-8c4d-5e6f7a8b9c0d\",\r\n \"Info\": \"Delete Outlook Mail Message: The operation failed. An object could not be found.\\r\\n\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The operation failed. An object could not be found. ---> System.Runtime.InteropServices.COMException: The operation failed. An object could not be found. (Exception from HRESULT: 0x8004010F)\",\r\n \"CreationTime\": \"2026-06-04T10:59:59.910Z\",\r\n \"ReleaseName\": \"InboxCleanupBot\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7042918,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T11:00:08.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"Id\": 793428713\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c4d5e6f7-8091-4b2c-9d3e-4f5a6b7c8d9e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:03.041Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] retrieved 24 messages from 'Inbox' at 11:00:03\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:03.120Z\",\r\n \"Message\": \"[Pick oldest message] target for deletion selected: EntryID 0000004A8F2C1B9D7E3F4A5B6C7D8E9F00112233; Subject 'Daily export - region EMEA'; received 2026-06-04T06:12:00Z; current location 'Inbox'\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:03.260Z\",\r\n \"Message\": \"[Process each message] Processing message: Daily export - region EMEA\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:04.910Z\",\r\n \"Message\": \"[Process each message] Processing message: Daily export - region APAC\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:06.380Z\",\r\n \"Message\": \"[Process each message] Processing message: Weekly reconciliation summary\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:07.840Z\",\r\n \"Message\": \"[Process each message] processed 24 of 24 messages in 00:00:04.7 (downstream API + attachment save per item)\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-04T11:00:08.060Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] target message (EntryID 0000004A8F2C1B9D7E3F4A5B6C7D8E9F00112233) not found in 'Inbox'; it appears to have been moved or deleted since retrieval at 11:00:03\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T11:00:08.150Z\",\r\n \"Message\": \"[Delete Outlook Mail Message] The operation failed. An object could not be found.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T11:00:08.198Z\",\r\n \"Message\": \"The operation failed. An object could not be found.\\r\\nin Main.xaml\\r\\n at DeleteOutlookMailMessage \\\"Delete Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T11:00:08.210Z\",\r\n \"Message\": \"Execution ended in: 00:00:08\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b3c4d5e6-7f80-4a1b-8c2d-3e4f5a6b7c8d\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the mailbox automation portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/task.yaml index 556b7b88f8..4d813bc5d1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-delete-outlook-mail-stale-reference/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/README.md new file mode 100644 index 0000000000..3d9cc49aea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/README.md @@ -0,0 +1,90 @@ +# Get Outlook Mail Messages - Cached Exchange Mode Desync (Branch 5) + +This scenario reproduces the **silent-miss** signature of the +`Get Outlook Mail Messages` failures playbook: a job that reports +**Success with no error** while the activity quietly returns only a +fraction of the Inbox. Recent emails the user can see in the Outlook +desktop client never make it into the export, and nothing in Orchestrator +flags a problem. + +## What this scenario uncovers + +**Root Cause:** The Robot's Outlook account runs in **Cached Exchange +Mode** with a **restricted Offline Settings window** ("Mail to keep +offline" set to a few months). Mail outside that window is never synced +into the local `.ost` cache. `Get Outlook Mail Messages` reads Outlook's +**local store** through COM, so it can only return cached mail - it cannot +see items the full online client shows. The job ends Successful, the +activity returns the cached subset, and the shortfall is invisible unless +you compare the retrieved count to the real Inbox size. + +The tell is in the full job logs: +`[Get Outlook Mail Messages] retrieved 142 messages from 'Inbox' +(Top=500, no filter); oldest item dated 2026-03-03` - the oldest returned +item sits at a ~3-month cache cutoff while `Top` is high and no `Filter` +narrows the result. + +This maps to: +`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` +(**Branch 5 - Cached Exchange Mode desync**). + +The user is framed as **off-host**, so the correct agent behavior is to +diagnose from Orchestrator + the workflow source, then hand the user a +host-side check (inspect/widen the Offline Settings slider to All, or +disable Cached Exchange Mode, and re-run; optionally move to Graph +o365-activities) **without fabricating host actions** - Cached Exchange +Mode state is only verifiable on the host. + +## Sibling-branch comparison (why this is Branch 5, not 1-4) + +| Branch | Signature | This scenario? | +|--------|-----------|----------------| +| **B1** Folder not resolved | `The specified folder does not exist` (nested/localized/shared-mailbox path) | No - job log shows `Inbox` resolved on the default account | +| **B2** Timeout on volume | `The operation has timed out` after `TimeoutMS`; job **Faulted** | No - job ended **Successful** in ~6 s | +| **B3** Outlook not running / COM | `The Outlook application is not running` / COM cast / hang | No - Outlook attached, read completed cleanly | +| **B4** Malformed `Filter` | `Filter` error, or zero results with a `Filter` set | No - **no `Filter`** is set; result is partial, not zero | +| **B5** Cached Exchange Mode desync | **No error**, but recent/older mail visible in the client is missing | **Yes** - this scenario | + +The defining trait of B5 is the **absence** of an error. The job is +`Successful`, the error-level logs are empty, and the only signal is the +shortfall + the oldest-item-at-cache-cutoff line in the full logs. An +agent that latches onto "Top too low" or "add a Filter" has misread the +case: `Top=500` is high and there is no Filter to fix - the limitation is +the host's cache window, not the workflow. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: `Get Outlook Mail Messages` with `MailFolder="Inbox"`, `Top="500"`, no `Filter` | + +The job discovery is modeled for a **SUCCESS-state** case: +`or jobs list ... --state Faulted` returns an **empty** list, while the +no-state and `--state Successful` variants return the one Successful job. +`or jobs get` reports a benign success `Info`; `or jobs logs ... --level +Error` is empty; the full `or jobs logs` carries the shortfall signal. + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +The judge **scores the conclusion, not the trajectory.** It grades the +agent's final response and its tool calls (did it Read the +mail-activities playbook), not internal investigation state. + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `get-outlook-mail-failures.md` Branch 5 (Cached Exchange + Mode desync). +- Agent recognized the **silent miss** (Successful, no error, fewer + messages than exist), **ruled out** the workflow-side causes (`Top` + high, no `Filter`, correct folder), and attributed the shortfall to + Cached Exchange Mode / a restricted Offline Settings window. +- Agent handed a **host-side check** (widen the Offline Settings slider to + All / disable Cached Exchange Mode and re-run; optionally move to Graph + o365-activities) **without fabricating host actions**. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/23530aca3d.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/23530aca3d.json deleted file mode 100644 index 6a18cfc2f4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/23530aca3d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e", - "StartTime": "2026-06-03T05:30:00.140Z", - "EndTime": "2026-06-03T05:30:06.420Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "ce5f6a70-8192-4940-5162-7a8b9c01d2e3", - "Info": "ComplianceMailExport completed. Messages retrieved: 142.", - "CreationTime": "2026-06-03T05:29:59.870Z", - "ReleaseName": "ComplianceMailExport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6029402, - "OrganizationUnitId": 7791618, - "FolderKey": "a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-03T05:30:06.420Z", - "JobError": {}, - "ErrorCode": "", - "TraceId": "b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e", - "Id": 682401993 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/4934670a0f.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/4934670a0f.json deleted file mode 100644 index ebcbe6a25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/4934670a0f.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the compliance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/README.md deleted file mode 100644 index dafe1136c2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/README.md +++ /dev/null @@ -1,89 +0,0 @@ -# Get Outlook Mail Messages - Cached Exchange Mode Desync (Branch 5) - -This scenario reproduces the **silent-miss** signature of the -`Get Outlook Mail Messages` failures playbook: a job that reports -**Success with no error** while the activity quietly returns only a -fraction of the Inbox. Recent emails the user can see in the Outlook -desktop client never make it into the export, and nothing in Orchestrator -flags a problem. - -## What this scenario uncovers - -**Root Cause:** The Robot's Outlook account runs in **Cached Exchange -Mode** with a **restricted Offline Settings window** ("Mail to keep -offline" set to a few months). Mail outside that window is never synced -into the local `.ost` cache. `Get Outlook Mail Messages` reads Outlook's -**local store** through COM, so it can only return cached mail - it cannot -see items the full online client shows. The job ends Successful, the -activity returns the cached subset, and the shortfall is invisible unless -you compare the retrieved count to the real Inbox size. - -The tell is in the full job logs: -`[Get Outlook Mail Messages] retrieved 142 messages from 'Inbox' -(Top=500, no filter); oldest item dated 2026-03-03` - the oldest returned -item sits at a ~3-month cache cutoff while `Top` is high and no `Filter` -narrows the result. - -This maps to: -`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` -(**Branch 5 - Cached Exchange Mode desync**). - -The user is framed as **off-host**, so the correct agent behavior is to -diagnose from Orchestrator + the workflow source, then hand the user a -host-side check (inspect/widen the Offline Settings slider to All, or -disable Cached Exchange Mode, and re-run; optionally move to Graph -o365-activities) **without fabricating host actions** - Cached Exchange -Mode state is only verifiable on the host. - -## Sibling-branch comparison (why this is Branch 5, not 1-4) - -| Branch | Signature | This scenario? | -|--------|-----------|----------------| -| **B1** Folder not resolved | `The specified folder does not exist` (nested/localized/shared-mailbox path) | No - job log shows `Inbox` resolved on the default account | -| **B2** Timeout on volume | `The operation has timed out` after `TimeoutMS`; job **Faulted** | No - job ended **Successful** in ~6 s | -| **B3** Outlook not running / COM | `The Outlook application is not running` / COM cast / hang | No - Outlook attached, read completed cleanly | -| **B4** Malformed `Filter` | `Filter` error, or zero results with a `Filter` set | No - **no `Filter`** is set; result is partial, not zero | -| **B5** Cached Exchange Mode desync | **No error**, but recent/older mail visible in the client is missing | **Yes** - this scenario | - -The defining trait of B5 is the **absence** of an error. The job is -`Successful`, the error-level logs are empty, and the only signal is the -shortfall + the oldest-item-at-cache-cutoff line in the full logs. An -agent that latches onto "Top too low" or "add a Filter" has misread the -case: `Top=500` is high and there is no Filter to fix - the limitation is -the host's cache window, not the workflow. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: `Get Outlook Mail Messages` with `MailFolder="Inbox"`, `Top="500"`, no `Filter` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the Branch 5 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table (quoted + unquoted UUID variants) | - -The job discovery is modeled for a **SUCCESS-state** case: -`or jobs list ... --state Faulted` returns an **empty** list, while the -no-state and `--state Successful` variants return the one Successful job. -`or jobs get` reports a benign success `Info`; `or jobs logs ... --level -Error` is empty; the full `or jobs logs` carries the shortfall signal. - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -The judge **scores the conclusion, not the trajectory.** It grades the -agent's final response and its tool calls (did it Read the -mail-activities playbook), not internal investigation state. - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `get-outlook-mail-failures.md` Branch 5 (Cached Exchange - Mode desync). -- Agent recognized the **silent miss** (Successful, no error, fewer - messages than exist), **ruled out** the workflow-side causes (`Top` - high, no `Filter`, correct folder), and attributed the shortfall to - Cached Exchange Mode / a restricted Offline Settings window. -- Agent handed a **host-side check** (widen the Offline Settings slider to - All / disable Cached Exchange Mode and re-run; optionally move to Graph - o365-activities) **without fabricating host actions**. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/eb539026d9.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/eb539026d9.json deleted file mode 100644 index af344e87d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/eb539026d9.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-03T05:30:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T05:30:01.640Z", - "Message": "[Get Outlook Mail Messages] Outlook desktop application attached; default profile for UIPATH\\ROBOTUSER1" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T05:30:02.110Z", - "Message": "[Get Outlook Mail Messages] resolved folder 'Inbox' on the default account" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T05:30:05.870Z", - "Message": "[Get Outlook Mail Messages] retrieved 142 messages from 'Inbox' (Top=500, no filter); oldest item dated 2026-03-03" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T05:30:06.020Z", - "Message": "[ComplianceMailExport] retrieved messages from Inbox" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T05:30:06.410Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/f726908bac.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/f726908bac.json deleted file mode 100644 index 2eea96cd85..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/f726908bac.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e", - "State": "Successful", - "ProcessName": "ComplianceMailExport", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T05:30:00.140Z", - "EndTime": "2026-06-03T05:30:06.420Z", - "CreationTime": "2026-06-03T05:29:59.870Z" - }, - { - "Key": "a4b5c6d7-8e9f-4a0b-1c2d-3e4f5a6b7c8d", - "State": "Successful", - "ProcessName": "ComplianceMailExport", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T05:30:00.100Z", - "EndTime": "2026-06-02T05:30:06.980Z", - "CreationTime": "2026-06-02T05:29:59.820Z" - }, - { - "Key": "93a4b5c6-7d8e-4f90-a1b2-c3d4e5f6a7b8", - "State": "Successful", - "ProcessName": "ComplianceMailExport", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-01T05:30:00.210Z", - "EndTime": "2026-06-01T05:30:07.330Z", - "CreationTime": "2026-06-01T05:29:59.910Z" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/manifest.json deleted file mode 100644 index 31ee010121..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Get Outlook Mail Messages scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "4934670a0f.json" - }, - { - "match": "or jobs list --folder-key \"a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d\" --state Successful --output json", - "file": "f726908bac.json" - }, - { - "match": "or jobs list --folder-key a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d --state Successful --output json", - "file": "f726908bac.json" - }, - { - "match": "or jobs list --folder-key \"a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d\" --output json", - "file": "f726908bac.json" - }, - { - "match": "or jobs list --folder-key a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d --output json", - "file": "f726908bac.json" - }, - { - "match": "or jobs get \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\" --output json", - "file": "23530aca3d.json" - }, - { - "match": "or jobs get b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e --output json", - "file": "23530aca3d.json" - }, - { - "match": "or jobs logs \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\" --output json", - "file": "eb539026d9.json" - }, - { - "match": "or jobs logs b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e --output json", - "file": "eb539026d9.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/uip-fixture.json new file mode 100644 index 0000000000..77cfe83f3c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ComplianceMailExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T05:30:00.140Z\",\r\n \"EndTime\": \"2026-06-03T05:30:06.420Z\",\r\n \"CreationTime\": \"2026-06-03T05:29:59.870Z\"\r\n },\r\n {\r\n \"Key\": \"a4b5c6d7-8e9f-4a0b-1c2d-3e4f5a6b7c8d\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ComplianceMailExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T05:30:00.100Z\",\r\n \"EndTime\": \"2026-06-02T05:30:06.980Z\",\r\n \"CreationTime\": \"2026-06-02T05:29:59.820Z\"\r\n },\r\n {\r\n \"Key\": \"93a4b5c6-7d8e-4f90-a1b2-c3d4e5f6a7b8\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ComplianceMailExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-01T05:30:00.210Z\",\r\n \"EndTime\": \"2026-06-01T05:30:07.330Z\",\r\n \"CreationTime\": \"2026-06-01T05:29:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ComplianceMailExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T05:30:00.140Z\",\r\n \"EndTime\": \"2026-06-03T05:30:06.420Z\",\r\n \"CreationTime\": \"2026-06-03T05:29:59.870Z\"\r\n },\r\n {\r\n \"Key\": \"a4b5c6d7-8e9f-4a0b-1c2d-3e4f5a6b7c8d\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ComplianceMailExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T05:30:00.100Z\",\r\n \"EndTime\": \"2026-06-02T05:30:06.980Z\",\r\n \"CreationTime\": \"2026-06-02T05:29:59.820Z\"\r\n },\r\n {\r\n \"Key\": \"93a4b5c6-7d8e-4f90-a1b2-c3d4e5f6a7b8\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ComplianceMailExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-01T05:30:00.210Z\",\r\n \"EndTime\": \"2026-06-01T05:30:07.330Z\",\r\n \"CreationTime\": \"2026-06-01T05:29:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"StartTime\": \"2026-06-03T05:30:00.140Z\",\r\n \"EndTime\": \"2026-06-03T05:30:06.420Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"ce5f6a70-8192-4940-5162-7a8b9c01d2e3\",\r\n \"Info\": \"ComplianceMailExport completed. Messages retrieved: 142.\",\r\n \"CreationTime\": \"2026-06-03T05:29:59.870Z\",\r\n \"ReleaseName\": \"ComplianceMailExport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6029402,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-03T05:30:06.420Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e\",\r\n \"Id\": 682401993\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b6c7d8e9-0f1a-4b2c-9d3e-4f5a6b7c8d9e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T05:30:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T05:30:01.640Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Outlook desktop application attached; default profile for UIPATH\\\\ROBOTUSER1\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T05:30:02.110Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] resolved folder 'Inbox' on the default account\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T05:30:05.870Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] retrieved 142 messages from 'Inbox' (Top=500, no filter); oldest item dated 2026-03-03\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T05:30:06.020Z\",\r\n \"Message\": \"[ComplianceMailExport] retrieved messages from Inbox\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T05:30:06.410Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the compliance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/task.yaml index 352ebd3bed..4310f874e2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-cached-mode-desync/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/README.md new file mode 100644 index 0000000000..5a39857a43 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/README.md @@ -0,0 +1,70 @@ +# Get Outlook Mail Messages Failure - Malformed Filter (DASL/Jet) + +This scenario reproduces a runtime `Get Outlook Mail Messages` failure caused +by a malformed DASL/Jet `Filter`. The `Filter` property holds +`[Subject] = VarSubject` -- the right-hand value is **unquoted**, so it is not +a valid DASL literal. When Outlook applies the restriction it cannot parse the +condition and the job faults with +`Cannot parse condition. Error at "VarSubject".`, surfaced through COM as +`0x80020009 (DISP_E_EXCEPTION)`. The folder resolves and the COM session is +fine -- the only fault is the filter syntax. + +## What this scenario uncovers + +**Root Cause:** The `Get Outlook Mail Messages` activity in `Main.xaml` runs +with `Filter="[Subject] = VarSubject"`. The value is unquoted, so Outlook's +DASL/Jet parser rejects the condition. The fix is to single-quote the value: +`[Subject] = 'Text to find'`, or with a variable +`"[Subject] = '" + mySubjectVariable + "'"`. + +This maps to +`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md`, +**Branch 4 (Malformed `Filter`)**. + +The job logs are the smoking gun: Outlook attaches over COM and the `Inbox` +folder resolves cleanly, then the activity echoes the applied restriction +`[Get Outlook Mail Messages] applying Filter: [Subject] = VarSubject` on the +line immediately before the parse error -- the unquoted value is visible. + +## Sibling-branch comparison + +This is Branch 4. The agent must NOT conclude any sibling branch (nor a null +variable): + +| Branch | Signature | Why it is NOT this scenario | +|---|---|---| +| 1 - Folder not resolved | `The specified folder does not exist` | Log shows `Inbox` resolved on the default profile | +| 2 - Timeout on large folder | `The operation has timed out` after `TimeoutMS` | Faulted in ~2 s with a parse error, not a timeout | +| 3 - Outlook not running / COM / privilege | `The Outlook application is not running` / cast failure / hang | Log shows Outlook attached over COM; no cast error, no hang | +| **4 - Malformed `Filter`** | `Cannot parse condition. Error at "..."` | **This scenario:** unquoted DASL value `[Subject] = VarSubject` | +| 5 - Cached Exchange Mode desync | NO error, mail silently missing | The job faulted with an explicit parse error | +| (not a branch) Null/empty subject variable | n/a | The bug is the missing quotes (DASL syntax), not a null value | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `Filter="[Subject] = VarSubject"` on the Get Outlook Mail Messages activity | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This scenario **scores the conclusion, not the trajectory.** The `llm_judge` +grades the agent's final response and tool calls against `RESOLUTION.md`; it +does not require a specific investigation path or any internal-state file. + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the `get-outlook-mail-failures` playbook, Branch 4. +- Agent attributed the fault to the unquoted value in the `Filter` + expression (`[Subject] = VarSubject`), citing `Main.xaml` and the job-log + applied-filter line. +- Agent recommended single-quoting the DASL value + (`[Subject] = 'Text'` / `"[Subject] = '" + var + "'"`) and did NOT conclude a + sibling branch or a null/empty subject variable. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/5a01cc6d46.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/5a01cc6d46.json deleted file mode 100644 index 727efa0ae0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/5a01cc6d46.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the support triage RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/95c5fa7694.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/95c5fa7694.json deleted file mode 100644 index a9cc8b8184..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/95c5fa7694.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-03T09:25:02.281Z", - "Message": "[Get Outlook Mail Messages] Cannot parse condition. Error at \"VarSubject\"." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T09:25:02.310Z", - "Message": "Cannot parse condition. Error at \"VarSubject\". (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/README.md deleted file mode 100644 index f8dbb15e98..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Get Outlook Mail Messages Failure - Malformed Filter (DASL/Jet) - -This scenario reproduces a runtime `Get Outlook Mail Messages` failure caused -by a malformed DASL/Jet `Filter`. The `Filter` property holds -`[Subject] = VarSubject` -- the right-hand value is **unquoted**, so it is not -a valid DASL literal. When Outlook applies the restriction it cannot parse the -condition and the job faults with -`Cannot parse condition. Error at "VarSubject".`, surfaced through COM as -`0x80020009 (DISP_E_EXCEPTION)`. The folder resolves and the COM session is -fine -- the only fault is the filter syntax. - -## What this scenario uncovers - -**Root Cause:** The `Get Outlook Mail Messages` activity in `Main.xaml` runs -with `Filter="[Subject] = VarSubject"`. The value is unquoted, so Outlook's -DASL/Jet parser rejects the condition. The fix is to single-quote the value: -`[Subject] = 'Text to find'`, or with a variable -`"[Subject] = '" + mySubjectVariable + "'"`. - -This maps to -`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md`, -**Branch 4 (Malformed `Filter`)**. - -The job logs are the smoking gun: Outlook attaches over COM and the `Inbox` -folder resolves cleanly, then the activity echoes the applied restriction -`[Get Outlook Mail Messages] applying Filter: [Subject] = VarSubject` on the -line immediately before the parse error -- the unquoted value is visible. - -## Sibling-branch comparison - -This is Branch 4. The agent must NOT conclude any sibling branch (nor a null -variable): - -| Branch | Signature | Why it is NOT this scenario | -|---|---|---| -| 1 - Folder not resolved | `The specified folder does not exist` | Log shows `Inbox` resolved on the default profile | -| 2 - Timeout on large folder | `The operation has timed out` after `TimeoutMS` | Faulted in ~2 s with a parse error, not a timeout | -| 3 - Outlook not running / COM / privilege | `The Outlook application is not running` / cast failure / hang | Log shows Outlook attached over COM; no cast error, no hang | -| **4 - Malformed `Filter`** | `Cannot parse condition. Error at "..."` | **This scenario:** unquoted DASL value `[Subject] = VarSubject` | -| 5 - Cached Exchange Mode desync | NO error, mail silently missing | The job faulted with an explicit parse error | -| (not a branch) Null/empty subject variable | n/a | The bug is the missing quotes (DASL syntax), not a null value | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `Filter="[Subject] = VarSubject"` on the Get Outlook Mail Messages activity | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 4 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This scenario **scores the conclusion, not the trajectory.** The `llm_judge` -grades the agent's final response and tool calls against `RESOLUTION.md`; it -does not require a specific investigation path or any internal-state file. - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the `get-outlook-mail-failures` playbook, Branch 4. -- Agent attributed the fault to the unquoted value in the `Filter` - expression (`[Subject] = VarSubject`), citing `Main.xaml` and the job-log - applied-filter line. -- Agent recommended single-quoting the DASL value - (`[Subject] = 'Text'` / `"[Subject] = '" + var + "'"`) and did NOT conclude a - sibling branch or a null/empty subject variable. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/bd507ac50a.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/bd507ac50a.json deleted file mode 100644 index 326a408fc0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/bd507ac50a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60", - "StartTime": "2026-06-03T09:25:00.140Z", - "EndTime": "2026-06-03T09:25:02.330Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "ae5f60d2-7182-4839-4051-6f7a8b90c1d2", - "Info": "Get Outlook Mail Messages: Cannot parse condition. Error at \"VarSubject\".\r\n\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Cannot parse condition. Error at \"VarSubject\". (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))", - "CreationTime": "2026-06-03T09:24:59.900Z", - "ReleaseName": "SupportTriageReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6018407, - "OrganizationUnitId": 7791618, - "FolderKey": "c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-03T09:25:02.330Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60", - "Id": 682319944 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/d7cff08234.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/d7cff08234.json deleted file mode 100644 index 78670972fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/d7cff08234.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60", - "State": "Faulted", - "ProcessName": "SupportTriageReader", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:25:00.140Z", - "EndTime": "2026-06-03T09:25:02.330Z", - "CreationTime": "2026-06-03T09:24:59.900Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/dc13dd8a74.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/dc13dd8a74.json deleted file mode 100644 index ff03a18329..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/dc13dd8a74.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-03T09:25:00.190Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T09:25:01.004Z", - "Message": "[Get Outlook Mail Messages] Attached to running Outlook application via COM" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T09:25:01.512Z", - "Message": "[Get Outlook Mail Messages] Resolved mail folder 'Inbox' on the default profile" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T09:25:02.118Z", - "Message": "[Get Outlook Mail Messages] applying Filter: [Subject] = VarSubject" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T09:25:02.281Z", - "Message": "[Get Outlook Mail Messages] Cannot parse condition. Error at \"VarSubject\"." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T09:25:02.310Z", - "Message": "Cannot parse condition. Error at \"VarSubject\". (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T09:25:02.325Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/manifest.json deleted file mode 100644 index 948ae8cfd5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Get Outlook Mail Messages scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "5a01cc6d46.json" - }, - { - "match": "or jobs list --folder-key \"c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f\" --state Faulted --output json", - "file": "d7cff08234.json" - }, - { - "match": "or jobs list --folder-key c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f --state Faulted --output json", - "file": "d7cff08234.json" - }, - { - "match": "or jobs get \"d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60\" --output json", - "file": "bd507ac50a.json" - }, - { - "match": "or jobs get d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60 --output json", - "file": "bd507ac50a.json" - }, - { - "match": "or jobs logs \"d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60\" --level Error --output json", - "file": "95c5fa7694.json" - }, - { - "match": "or jobs logs d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60 --level Error --output json", - "file": "95c5fa7694.json" - }, - { - "match": "or jobs logs \"d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60\" --output json", - "file": "dc13dd8a74.json" - }, - { - "match": "or jobs logs d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60 --output json", - "file": "dc13dd8a74.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/uip-fixture.json new file mode 100644 index 0000000000..d77bfc8e49 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportTriageReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:25:00.140Z\",\r\n \"EndTime\": \"2026-06-03T09:25:02.330Z\",\r\n \"CreationTime\": \"2026-06-03T09:24:59.900Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T09:25:02.281Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Cannot parse condition. Error at \\\"VarSubject\\\".\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T09:25:02.310Z\",\r\n \"Message\": \"Cannot parse condition. Error at \\\"VarSubject\\\". (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60\",\r\n \"StartTime\": \"2026-06-03T09:25:00.140Z\",\r\n \"EndTime\": \"2026-06-03T09:25:02.330Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"ae5f60d2-7182-4839-4051-6f7a8b90c1d2\",\r\n \"Info\": \"Get Outlook Mail Messages: Cannot parse condition. Error at \\\"VarSubject\\\".\\r\\n\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Cannot parse condition. Error at \\\"VarSubject\\\". (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))\",\r\n \"CreationTime\": \"2026-06-03T09:24:59.900Z\",\r\n \"ReleaseName\": \"SupportTriageReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6018407,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-03T09:25:02.330Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60\",\r\n \"Id\": 682319944\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d2e3f4a5-6b7c-4d8e-9f0a-1b2c3d4e5f60", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T09:25:00.190Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T09:25:01.004Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Attached to running Outlook application via COM\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T09:25:01.512Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Resolved mail folder 'Inbox' on the default profile\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T09:25:02.118Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] applying Filter: [Subject] = VarSubject\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T09:25:02.281Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Cannot parse condition. Error at \\\"VarSubject\\\".\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T09:25:02.310Z\",\r\n \"Message\": \"Cannot parse condition. Error at \\\"VarSubject\\\". (Exception from HRESULT: 0x80020009 (DISP_E_EXCEPTION))\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T09:25:02.325Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c1d2e3f4-5a6b-4c7d-8e9f-0a1b2c3d4e5f\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the support triage RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/task.yaml index 86c06e2cf0..577c90aa8b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-filter-syntax/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/README.md new file mode 100644 index 0000000000..bed1b3e8c1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/README.md @@ -0,0 +1,72 @@ +# Get Outlook Mail Messages Failure - Folder Not Resolved (Branch 1) + +This scenario reproduces a runtime `Get Outlook Mail Messages` failure +where the configured `MailFolder` cannot be resolved. The activity is +set to read the nested folder `Inbox\Invoices` using backslash +nested-path syntax, which does not resolve to a real Outlook folder on +the project's Mail-package version (`UiPath.Mail.Activities [1.18.3]`). +The job faults with `The specified folder does not exist.` + +## What this scenario uncovers + +**Root Cause:** `Main.xaml` configures the `Get Outlook Mail Messages` +activity with `MailFolder="Inbox\Invoices"`. The Outlook COM session is +healthy -- the activity attaches to the running Outlook instance under +the default profile -- but the backslash nested-path string does not +resolve, so the activity raises `The specified folder does not exist.` +The fix is to address the subfolder with the absolute path form +`\\\Inbox\Invoices`, or to target the top-level `Inbox`, or to +set the `Account` field when the subfolder lives under a shared mailbox. + +This maps to: +`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` +(Branch 1 -- Folder not resolved). + +This is a **workflow-fixable configuration bug**: the fix is a property +change in `Main.xaml`, not a host-side action. The user has both +Orchestrator access and the project source, so the agent can diagnose +end-to-end from Orchestrator + the workflow source without host data. + +## Sibling-branch comparison + +The Get Outlook Mail playbook has five cause-branches. This scenario is +Branch 1. What distinguishes it from the others: + +| Branch | Signature | This scenario? | Why not | +|--------|-----------|----------------|---------| +| 1 Folder not resolved | `The specified folder does not exist.` | **YES** | `MailFolder="Inbox\Invoices"` backslash nested path fails to resolve | +| 2 Timeout on large folder | `The operation has timed out` after `TimeoutMS` | No | Job ran ~2s; error is folder-not-found, not a timeout; enumeration never started | +| 3 Outlook not running / COM / privilege | `The Outlook application is not running` / COM cast / a freeze | No | Logs show a clean attach to the running Outlook instance; no COM error, no hang | +| 4 Malformed `Filter` | `Filter` error or zero-results-with-a-filter | No | No `Filter` is set; error is folder resolution, not DASL/Jet syntax | +| 5 Cached Exchange Mode desync | No error, but mail visible in the client is missing | No | The job faults with an explicit exception; this is not a silent miss | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `MailFolder="Inbox\Invoices"` on the Get Outlook Mail Messages activity | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook Branch 1 signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This scenario **scores the conclusion, not the trajectory.** The +`llm_judge` grades only the agent's final response and tool calls +against `RESOLUTION.md`; it does not inspect internal investigation +state. Passing requires: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `get-outlook-mail-failures.md` Branch 1 (folder not + resolved). +- Agent attributed the fault to the `MailFolder="Inbox\Invoices"` + backslash nested-path string that does not resolve (not a timeout, + COM/Outlook-not-running, `Filter`, or Cached Exchange Mode cause). +- Agent recommended the absolute-path form + (`\\\Inbox\Invoices`), or targeting the top-level folder, or + setting `Account` for a shared mailbox. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/64a65f747b.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/64a65f747b.json deleted file mode 100644 index a1bf720e16..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/64a65f747b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234", - "State": "Faulted", - "ProcessName": "InvoiceMailReader", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T07:10:00.140Z", - "EndTime": "2026-06-03T07:10:02.260Z", - "CreationTime": "2026-06-03T07:09:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/README.md deleted file mode 100644 index d3fd1b71bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/README.md +++ /dev/null @@ -1,71 +0,0 @@ -# Get Outlook Mail Messages Failure - Folder Not Resolved (Branch 1) - -This scenario reproduces a runtime `Get Outlook Mail Messages` failure -where the configured `MailFolder` cannot be resolved. The activity is -set to read the nested folder `Inbox\Invoices` using backslash -nested-path syntax, which does not resolve to a real Outlook folder on -the project's Mail-package version (`UiPath.Mail.Activities [1.18.3]`). -The job faults with `The specified folder does not exist.` - -## What this scenario uncovers - -**Root Cause:** `Main.xaml` configures the `Get Outlook Mail Messages` -activity with `MailFolder="Inbox\Invoices"`. The Outlook COM session is -healthy -- the activity attaches to the running Outlook instance under -the default profile -- but the backslash nested-path string does not -resolve, so the activity raises `The specified folder does not exist.` -The fix is to address the subfolder with the absolute path form -`\\\Inbox\Invoices`, or to target the top-level `Inbox`, or to -set the `Account` field when the subfolder lives under a shared mailbox. - -This maps to: -`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` -(Branch 1 -- Folder not resolved). - -This is a **workflow-fixable configuration bug**: the fix is a property -change in `Main.xaml`, not a host-side action. The user has both -Orchestrator access and the project source, so the agent can diagnose -end-to-end from Orchestrator + the workflow source without host data. - -## Sibling-branch comparison - -The Get Outlook Mail playbook has five cause-branches. This scenario is -Branch 1. What distinguishes it from the others: - -| Branch | Signature | This scenario? | Why not | -|--------|-----------|----------------|---------| -| 1 Folder not resolved | `The specified folder does not exist.` | **YES** | `MailFolder="Inbox\Invoices"` backslash nested path fails to resolve | -| 2 Timeout on large folder | `The operation has timed out` after `TimeoutMS` | No | Job ran ~2s; error is folder-not-found, not a timeout; enumeration never started | -| 3 Outlook not running / COM / privilege | `The Outlook application is not running` / COM cast / a freeze | No | Logs show a clean attach to the running Outlook instance; no COM error, no hang | -| 4 Malformed `Filter` | `Filter` error or zero-results-with-a-filter | No | No `Filter` is set; error is folder resolution, not DASL/Jet syntax | -| 5 Cached Exchange Mode desync | No error, but mail visible in the client is missing | No | The job faults with an explicit exception; this is not a silent miss | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `MailFolder="Inbox\Invoices"` on the Get Outlook Mail Messages activity | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 1 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook Branch 1 signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This scenario **scores the conclusion, not the trajectory.** The -`llm_judge` grades only the agent's final response and tool calls -against `RESOLUTION.md`; it does not inspect internal investigation -state. Passing requires: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `get-outlook-mail-failures.md` Branch 1 (folder not - resolved). -- Agent attributed the fault to the `MailFolder="Inbox\Invoices"` - backslash nested-path string that does not resolve (not a timeout, - COM/Outlook-not-running, `Filter`, or Cached Exchange Mode cause). -- Agent recommended the absolute-path form - (`\\\Inbox\Invoices`), or targeting the top-level folder, or - setting `Account` for a shared mailbox. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/c858d01983.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/c858d01983.json deleted file mode 100644 index 885462a9d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/c858d01983.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234", - "StartTime": "2026-06-03T07:10:00.140Z", - "EndTime": "2026-06-03T07:10:02.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "a9b0c1d2-3e4f-4a5b-9c6d-7e8f90123456", - "Info": "Get Outlook Mail Messages: The specified folder does not exist.\r\n\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.", - "CreationTime": "2026-06-03T07:09:59.910Z", - "ReleaseName": "InvoiceMailReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021407, - "OrganizationUnitId": 7791618, - "FolderKey": "e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-03T07:10:02.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234", - "Id": 682401773 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/cd0d557ab4.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/cd0d557ab4.json deleted file mode 100644 index a699249c13..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/cd0d557ab4.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/e837ca1cc9.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/e837ca1cc9.json deleted file mode 100644 index 99fda66b30..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/e837ca1cc9.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-03T07:10:02.201Z", - "Message": "[Get Outlook Mail Messages] could not resolve folder 'Inbox\\Invoices' under the default profile" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T07:10:02.244Z", - "Message": "[Get Outlook Mail Messages] The specified folder does not exist." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T07:10:02.252Z", - "Message": "The specified folder does not exist.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/e8914b3f77.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/e8914b3f77.json deleted file mode 100644 index 9f2805a677..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/e8914b3f77.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-03T07:10:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T07:10:01.402Z", - "Message": "[Get Outlook Mail Messages] Attached to the running Outlook instance for the default profile" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T07:10:01.961Z", - "Message": "[Get Outlook Mail Messages] resolving folder path 'Inbox\\Invoices' against the default profile" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T07:10:02.201Z", - "Message": "[Get Outlook Mail Messages] could not resolve folder 'Inbox\\Invoices' under the default profile" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T07:10:02.244Z", - "Message": "[Get Outlook Mail Messages] The specified folder does not exist." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T07:10:02.252Z", - "Message": "The specified folder does not exist.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T07:10:02.255Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/manifest.json deleted file mode 100644 index d8a6d3d523..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Get Outlook Mail Messages scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "cd0d557ab4.json" - }, - { - "match": "or jobs list --folder-key \"e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012\" --state Faulted --output json", - "file": "64a65f747b.json" - }, - { - "match": "or jobs list --folder-key e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012 --state Faulted --output json", - "file": "64a65f747b.json" - }, - { - "match": "or jobs get \"f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234\" --output json", - "file": "c858d01983.json" - }, - { - "match": "or jobs get f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234 --output json", - "file": "c858d01983.json" - }, - { - "match": "or jobs logs \"f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234\" --level Error --output json", - "file": "e837ca1cc9.json" - }, - { - "match": "or jobs logs f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234 --level Error --output json", - "file": "e837ca1cc9.json" - }, - { - "match": "or jobs logs \"f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234\" --output json", - "file": "e8914b3f77.json" - }, - { - "match": "or jobs logs f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234 --output json", - "file": "e8914b3f77.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..49b8bc02c6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceMailReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T07:10:00.140Z\",\r\n \"EndTime\": \"2026-06-03T07:10:02.260Z\",\r\n \"CreationTime\": \"2026-06-03T07:09:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.201Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] could not resolve folder 'Inbox\\\\Invoices' under the default profile\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.244Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] The specified folder does not exist.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.252Z\",\r\n \"Message\": \"The specified folder does not exist.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234\",\r\n \"StartTime\": \"2026-06-03T07:10:00.140Z\",\r\n \"EndTime\": \"2026-06-03T07:10:02.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"a9b0c1d2-3e4f-4a5b-9c6d-7e8f90123456\",\r\n \"Info\": \"Get Outlook Mail Messages: The specified folder does not exist.\\r\\n\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.\",\r\n \"CreationTime\": \"2026-06-03T07:09:59.910Z\",\r\n \"ReleaseName\": \"InvoiceMailReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021407,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-03T07:10:02.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234\",\r\n \"Id\": 682401773\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f8a9b0c1-2d3e-4f4a-9b5c-6d7e8f901234", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T07:10:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T07:10:01.402Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Attached to the running Outlook instance for the default profile\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T07:10:01.961Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] resolving folder path 'Inbox\\\\Invoices' against the default profile\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.201Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] could not resolve folder 'Inbox\\\\Invoices' under the default profile\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.244Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] The specified folder does not exist.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.252Z\",\r\n \"Message\": \"The specified folder does not exist.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T07:10:02.255Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e7f8a9b0-1c2d-4e3f-8a4b-5c6d7e8f9012\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/task.yaml index b3d6c80ab5..72be5b96cb 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-folder-not-found/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/README.md new file mode 100644 index 0000000000..833c55e21d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/README.md @@ -0,0 +1,79 @@ +# Get Outlook Mail Messages Failure - Outlook Not Running / Broken COM Session (Branch 3) + +This scenario reproduces a runtime `Get Outlook Mail Messages` failure on +an **unattended** Robot where the desktop Outlook **COM session is not +available**: there is no running `OUTLOOK.EXE` instance for the Robot's +Windows user (and/or UiPath and Outlook run at mismatched privilege +levels). The COM layer surfaces this as +`The Outlook application is not running.` with an inner +`System.Runtime.InteropServices.COMException: The RPC server is +unavailable. (Exception from HRESULT: 0x800706BA)`. + +## What this scenario uncovers + +**Root Cause:** The `Get Outlook Mail Messages` activity in `Main.xaml` +attaches to the desktop Outlook through COM interop, but the job runs +**Unattended** on MOCK-HOST as `UIPATH\ROBOTUSER1` with no interactive +desktop and no running Outlook to attach to (and/or a privilege mismatch +blocking the cross-process call). The `MailFolder="Inbox"` and the (absent) +`Filter` are fine -- the failure is the COM session, not folder +resolution. + +This maps to: +`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` +(**Branch 3 -- Outlook not running / COM session broken / privilege +mismatch**). + +The fix is **host-side**, and the user is framed as **off-host** (only +Orchestrator access). The correct agent behavior is to hand a host-side +check list (ensure Outlook running / `Start Process outlook.exe` before +the activity / run UiPath and Outlook at the same privilege level / +migrate the unattended read to the modern Graph o365-activities) **without +fabricating host actions** it cannot perform. + +## Sibling-branch comparison (Get Outlook Mail Messages) + +| Branch | Signature | Cause | This scenario? | +|---|---|---|---| +| 1 - Folder not resolved | `The specified folder does not exist.` | `MailFolder` / `Account` string is wrong (nested backslash path, shared mailbox, localized name) | No - `MailFolder="Inbox"` resolves | +| 2 - Timeout on large folder | `The operation has timed out` after `TimeoutMS` | Folder too big / attachments huge vs the timeout | No - faulted ~4 s with an explicit COM error, not a timeout | +| **3 - Outlook not running / COM session / privilege** | **`The Outlook application is not running.` / COM cast / RPC server unavailable / freeze** | **No running OUTLOOK.EXE for the Robot user on an unattended host, and/or a UiPath-vs-Outlook privilege mismatch** | **Yes** | +| 4 - Malformed `Filter` | `Filter` error or zero results with a filter set | Unquoted / non-DASL/Jet `Filter` value | No - no `Filter` is set | +| 5 - Cached Exchange Mode desync | **No error**, missing recent/old mail | Offline cache window too narrow | No - the job faulted with a hard error | + +> **Distinct from the Send Outlook Mail install surface.** A COM +> **library-not-registered / bitness** cast error +> (`REGDB_E_CLASSNOTREG` / `Unable to cast COM object`) attributed to the +> Outlook **install** is the `send-outlook-mail-failures.md` Branch 1 +> surface. Here Outlook **is** installed -- it is simply **not running** / +> blocked by privilege. The scenario keeps these distinct. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: a `Get Outlook Mail Messages` activity with `MailFolder="Inbox"` (valid), unattended `project.json` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. The smoking gun is in the full job +> logs: an Info that the run is Unattended with no interactive desktop, and +> a line that no running Outlook instance was found for the Robot's user. + +## Success criteria + +This test **scores the conclusion, not the trajectory** -- two criteria: + +- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). +- Agent reached the same conclusion as `RESOLUTION.md` (`llm_judge`, + presentation-graded): matched `get-outlook-mail-failures` Branch 3, named + the Outlook-not-running / broken COM session failure, attributed it to no + running Outlook for the unattended Robot user (and/or a privilege + mismatch) -- not a folder / filter / timeout / registration issue -- and + handed a host check list (ensure Outlook running / `Start Process + outlook.exe` / same privilege level / Graph o365 for unattended) without + fabricating host actions. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/2aea242132.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/2aea242132.json deleted file mode 100644 index 85d3c93be9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/2aea242132.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c", - "StartTime": "2026-06-03T04:00:00.090Z", - "EndTime": "2026-06-03T04:00:04.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d", - "Info": "Get Outlook Mail Messages: The Outlook application is not running.\r\n\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The Outlook application is not running. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)", - "CreationTime": "2026-06-03T03:59:59.870Z", - "ReleaseName": "QueueMailIntake", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 7044219, - "OrganizationUnitId": 8810427, - "FolderKey": "e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-03T04:00:04.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c", - "Id": 793428713 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/README.md deleted file mode 100644 index 9d11d41847..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# Get Outlook Mail Messages Failure - Outlook Not Running / Broken COM Session (Branch 3) - -This scenario reproduces a runtime `Get Outlook Mail Messages` failure on -an **unattended** Robot where the desktop Outlook **COM session is not -available**: there is no running `OUTLOOK.EXE` instance for the Robot's -Windows user (and/or UiPath and Outlook run at mismatched privilege -levels). The COM layer surfaces this as -`The Outlook application is not running.` with an inner -`System.Runtime.InteropServices.COMException: The RPC server is -unavailable. (Exception from HRESULT: 0x800706BA)`. - -## What this scenario uncovers - -**Root Cause:** The `Get Outlook Mail Messages` activity in `Main.xaml` -attaches to the desktop Outlook through COM interop, but the job runs -**Unattended** on MOCK-HOST as `UIPATH\ROBOTUSER1` with no interactive -desktop and no running Outlook to attach to (and/or a privilege mismatch -blocking the cross-process call). The `MailFolder="Inbox"` and the (absent) -`Filter` are fine -- the failure is the COM session, not folder -resolution. - -This maps to: -`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` -(**Branch 3 -- Outlook not running / COM session broken / privilege -mismatch**). - -The fix is **host-side**, and the user is framed as **off-host** (only -Orchestrator access). The correct agent behavior is to hand a host-side -check list (ensure Outlook running / `Start Process outlook.exe` before -the activity / run UiPath and Outlook at the same privilege level / -migrate the unattended read to the modern Graph o365-activities) **without -fabricating host actions** it cannot perform. - -## Sibling-branch comparison (Get Outlook Mail Messages) - -| Branch | Signature | Cause | This scenario? | -|---|---|---|---| -| 1 - Folder not resolved | `The specified folder does not exist.` | `MailFolder` / `Account` string is wrong (nested backslash path, shared mailbox, localized name) | No - `MailFolder="Inbox"` resolves | -| 2 - Timeout on large folder | `The operation has timed out` after `TimeoutMS` | Folder too big / attachments huge vs the timeout | No - faulted ~4 s with an explicit COM error, not a timeout | -| **3 - Outlook not running / COM session / privilege** | **`The Outlook application is not running.` / COM cast / RPC server unavailable / freeze** | **No running OUTLOOK.EXE for the Robot user on an unattended host, and/or a UiPath-vs-Outlook privilege mismatch** | **Yes** | -| 4 - Malformed `Filter` | `Filter` error or zero results with a filter set | Unquoted / non-DASL/Jet `Filter` value | No - no `Filter` is set | -| 5 - Cached Exchange Mode desync | **No error**, missing recent/old mail | Offline cache window too narrow | No - the job faulted with a hard error | - -> **Distinct from the Send Outlook Mail install surface.** A COM -> **library-not-registered / bitness** cast error -> (`REGDB_E_CLASSNOTREG` / `Unable to cast COM object`) attributed to the -> Outlook **install** is the `send-outlook-mail-failures.md` Branch 1 -> surface. Here Outlook **is** installed -- it is simply **not running** / -> blocked by privilege. The scenario keeps these distinct. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: a `Get Outlook Mail Messages` activity with `MailFolder="Inbox"` (valid), unattended `project.json` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 3 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. The smoking gun is in the full job -> logs: an Info that the run is Unattended with no interactive desktop, and -> a line that no running Outlook instance was found for the Robot's user. - -## Success criteria - -This test **scores the conclusion, not the trajectory** -- two criteria: - -- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). -- Agent reached the same conclusion as `RESOLUTION.md` (`llm_judge`, - presentation-graded): matched `get-outlook-mail-failures` Branch 3, named - the Outlook-not-running / broken COM session failure, attributed it to no - running Outlook for the unattended Robot user (and/or a privilege - mismatch) -- not a folder / filter / timeout / registration issue -- and - handed a host check list (ensure Outlook running / `Start Process - outlook.exe` / same privilege level / Graph o365 for unattended) without - fabricating host actions. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/aa17d0af19.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/aa17d0af19.json deleted file mode 100644 index 1cf4c2350d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/aa17d0af19.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-03T04:00:04.061Z", - "Message": "[Get Outlook Mail Messages] The Outlook application is not running." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T04:00:04.140Z", - "Message": "Get Outlook Mail Messages: The Outlook application is not running.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\nUiPath.Mail.Exception: The Outlook application is not running. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/b9c7416e67.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/b9c7416e67.json deleted file mode 100644 index 8c748acd7d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/b9c7416e67.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c", - "State": "Faulted", - "ProcessName": "QueueMailIntake", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T04:00:00.090Z", - "EndTime": "2026-06-03T04:00:04.220Z", - "CreationTime": "2026-06-03T03:59:59.870Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/e0b695e59c.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/e0b695e59c.json deleted file mode 100644 index 57f3ce92ef..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/e0b695e59c.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the queue-intake RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c4d5e6f7-8a9b-4c0d-9e1f-2a3b4c5d6e7f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/fdc68d1712.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/fdc68d1712.json deleted file mode 100644 index e51ca8b0e7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/fdc68d1712.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-03T04:00:00.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T04:00:00.305Z", - "Message": "Robot session is running Unattended (no interactive desktop) as UIPATH\\ROBOTUSER1 on MOCK-HOST" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T04:00:01.882Z", - "Message": "[Get Outlook Mail Messages] no running Outlook instance detected for the current Windows session; attempting to attach failed" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-03T04:00:03.410Z", - "Message": "[Get Outlook Mail Messages] could not bind to the Outlook COM server (OUTLOOK.EXE not running for UIPATH\\ROBOTUSER1); verify Outlook is started and runs at the same privilege level as the Robot" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T04:00:04.061Z", - "Message": "[Get Outlook Mail Messages] The Outlook application is not running." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T04:00:04.140Z", - "Message": "Get Outlook Mail Messages: The Outlook application is not running.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\nUiPath.Mail.Exception: The Outlook application is not running. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T04:00:04.205Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/manifest.json deleted file mode 100644 index 7de2468a34..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Get Outlook Mail Messages scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e0b695e59c.json" - }, - { - "match": "or jobs list --folder-key \"e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b\" --state Faulted --output json", - "file": "b9c7416e67.json" - }, - { - "match": "or jobs list --folder-key e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b --state Faulted --output json", - "file": "b9c7416e67.json" - }, - { - "match": "or jobs get \"f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c\" --output json", - "file": "2aea242132.json" - }, - { - "match": "or jobs get f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c --output json", - "file": "2aea242132.json" - }, - { - "match": "or jobs logs \"f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c\" --level Error --output json", - "file": "aa17d0af19.json" - }, - { - "match": "or jobs logs f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c --level Error --output json", - "file": "aa17d0af19.json" - }, - { - "match": "or jobs logs \"f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c\" --output json", - "file": "fdc68d1712.json" - }, - { - "match": "or jobs logs f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c --output json", - "file": "fdc68d1712.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/uip-fixture.json new file mode 100644 index 0000000000..ced9911c56 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"QueueMailIntake\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T04:00:00.090Z\",\r\n \"EndTime\": \"2026-06-03T04:00:04.220Z\",\r\n \"CreationTime\": \"2026-06-03T03:59:59.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T04:00:04.061Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] The Outlook application is not running.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T04:00:04.140Z\",\r\n \"Message\": \"Get Outlook Mail Messages: The Outlook application is not running.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\nUiPath.Mail.Exception: The Outlook application is not running. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c\",\r\n \"StartTime\": \"2026-06-03T04:00:00.090Z\",\r\n \"EndTime\": \"2026-06-03T04:00:04.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"a5b6c7d8-9e0f-4a1b-8c2d-3e4f5a6b7c8d\",\r\n \"Info\": \"Get Outlook Mail Messages: The Outlook application is not running.\\r\\n\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The Outlook application is not running. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)\",\r\n \"CreationTime\": \"2026-06-03T03:59:59.870Z\",\r\n \"ReleaseName\": \"QueueMailIntake\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 7044219,\r\n \"OrganizationUnitId\": 8810427,\r\n \"FolderKey\": \"e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-03T04:00:04.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c\",\r\n \"Id\": 793428713\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f4a5b6c7-8d9e-4f0a-9b1c-2d3e4f5a6b7c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T04:00:00.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T04:00:00.305Z\",\r\n \"Message\": \"Robot session is running Unattended (no interactive desktop) as UIPATH\\\\ROBOTUSER1 on MOCK-HOST\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T04:00:01.882Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] no running Outlook instance detected for the current Windows session; attempting to attach failed\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-03T04:00:03.410Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] could not bind to the Outlook COM server (OUTLOOK.EXE not running for UIPATH\\\\ROBOTUSER1); verify Outlook is started and runs at the same privilege level as the Robot\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T04:00:04.061Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] The Outlook application is not running.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T04:00:04.140Z\",\r\n \"Message\": \"Get Outlook Mail Messages: The Outlook application is not running.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\nUiPath.Mail.Exception: The Outlook application is not running. ---> System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T04:00:04.205Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e3f4a5b6-7c8d-4e9f-8a0b-1c2d3e4f5a6b\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the queue-intake RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c4d5e6f7-8a9b-4c0d-9e1f-2a3b4c5d6e7f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/task.yaml index e3a88947a3..4ce8b94c0c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-outlook-not-running/task.yaml @@ -27,22 +27,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/README.md new file mode 100644 index 0000000000..642c455762 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/README.md @@ -0,0 +1,64 @@ +# Get Outlook Mail Messages — Shared Mailbox / Account Omission (Branch 1) + +Runtime troubleshooting scenario for `UiPath.Mail.Outlook.Activities` `Get Outlook Mail Messages` +(`GetOutlookMailMessages`), covering the **shared-mailbox / `Account` omission** sub-cause of +`get-outlook-mail-failures.md` **Branch 1 (Folder not resolved)**. + +## What this scenario exercises + +An unattended job faults with `UiPath.Mail.Exception: The specified folder does not exist`. The +`MailFolder` (`Invoices`) is a plain, correct folder name — but it lives in a **shared mailbox** +(`accounts-payable@test.com`), and the activity's **`Account` field is blank**, so the lookup is scoped +to the Robot's default profile store where `Invoices` does not exist. The agent must attribute the +failure to the missing `Account` (shared-mailbox scope) and prescribe setting +`Account = "accounts-payable@test.com"` — NOT changing the folder string, and NOT a COM/session, +timeout, or filter cause. + +## The discriminator vs the folder-not-found sibling + +Branch 1 (Folder not resolved) has multiple sub-causes. This scenario is distinguished from +`mail-get-outlook-mail-folder-not-found` (which pins the **nested backslash path** `Inbox\Invoices`) by: + +- `MailFolder` is a **plain top-level name** (`Invoices`) — no backslash, no localization; the folder + genuinely exists. +- Source has `Account=""`, and the job logs show `No Account specified; resolving folder 'Invoices' + against the default profile store (robotuser1@test.com)` → `Could not find folder ...; stores available + to this profile: robotuser1@test.com, accounts-payable@test.com`. + +The smoking gun is that the shared `accounts-payable@test.com` store IS attached to the profile but was +never searched, because `Account` was blank — the fix is `Account`, not the folder string. + +## Sibling comparison (same playbook, Branch 1) + +| Sub-cause | Signature | This scenario? | +|---|---|---| +| Nested backslash path (`Inbox\Invoices`) | `folder does not exist`; resolved against default profile | No — `mail-get-outlook-mail-folder-not-found` | +| **Shared mailbox, `Account` blank** | **`folder does not exist`; folder is in a shared store the lookup never searched** | **Yes** | +| Localized folder name (e.g. `Posteingang`) | `folder does not exist`; name mismatches UI language | Not yet covered | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `GetOutlookMailMessages` with `Account=""`, `MailFolder="Invoices"`, no `Filter` | + +Diagnosis is not disclosed in any agent-visible name (project `PayablesInvoiceReader`, activity "Get +Outlook Mail Messages"). The prompt states observed symptoms only: the read fails saying the folder does +not exist, yet the folder is definitely present — in the Accounts Payable **shared** mailbox the user +has open alongside their own inbox. + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent attributed the failure to the shared-mailbox folder being read with a blank `Account` (default- + profile-only scope), and prescribed setting `Account` to the shared mailbox's SMTP address — not a + folder-string edit, COM/session, timeout, or filter fix. + +Playbook: `references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` +(Branch 1 — shared-mailbox / `Account` omission sub-cause). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/464db476bb.json deleted file mode 100644 index d8fc1b48bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/464db476bb.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3d4e5f6-7081-4a92-9b3c-4d5e6f708293", - "StartTime": "2026-06-04T07:10:00.150Z", - "EndTime": "2026-06-04T07:10:02.410Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "d4e5f6a7-8192-4b03-9c4d-5e6f70819304", - "Info": "Get Outlook Mail Messages: The specified folder does not exist.\r\n\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.", - "CreationTime": "2026-06-04T07:09:59.900Z", - "ReleaseName": "PayablesInvoiceReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6041407, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c3d4e5-6f70-4a81-9b2c-3d4e5f607182", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T07:10:02.410Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3d4e5f6-7081-4a92-9b3c-4d5e6f708293", - "Id": 682511884 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/6059da6ae1.json deleted file mode 100644 index 17ef46dc00..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c3d4e5f6-7081-4a92-9b3c-4d5e6f708293", - "State": "Faulted", - "ProcessName": "PayablesInvoiceReader", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T07:10:00.150Z", - "EndTime": "2026-06-04T07:10:02.410Z", - "CreationTime": "2026-06-04T07:09:59.900Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/78f94b6048.json deleted file mode 100644 index 183b89d86e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/78f94b6048.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c3d4e5-6f70-4a81-9b2c-3d4e5f607182", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/8f63dd2da5.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/8f63dd2da5.json deleted file mode 100644 index da127197b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/8f63dd2da5.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T07:10:02.360Z", - "Message": "Get Outlook Mail Messages: The specified folder does not exist.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/README.md deleted file mode 100644 index 1a7a143a1c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# Get Outlook Mail Messages — Shared Mailbox / Account Omission (Branch 1) - -Runtime troubleshooting scenario for `UiPath.Mail.Outlook.Activities` `Get Outlook Mail Messages` -(`GetOutlookMailMessages`), covering the **shared-mailbox / `Account` omission** sub-cause of -`get-outlook-mail-failures.md` **Branch 1 (Folder not resolved)**. - -## What this scenario exercises - -An unattended job faults with `UiPath.Mail.Exception: The specified folder does not exist`. The -`MailFolder` (`Invoices`) is a plain, correct folder name — but it lives in a **shared mailbox** -(`accounts-payable@test.com`), and the activity's **`Account` field is blank**, so the lookup is scoped -to the Robot's default profile store where `Invoices` does not exist. The agent must attribute the -failure to the missing `Account` (shared-mailbox scope) and prescribe setting -`Account = "accounts-payable@test.com"` — NOT changing the folder string, and NOT a COM/session, -timeout, or filter cause. - -## The discriminator vs the folder-not-found sibling - -Branch 1 (Folder not resolved) has multiple sub-causes. This scenario is distinguished from -`mail-get-outlook-mail-folder-not-found` (which pins the **nested backslash path** `Inbox\Invoices`) by: - -- `MailFolder` is a **plain top-level name** (`Invoices`) — no backslash, no localization; the folder - genuinely exists. -- Source has `Account=""`, and the job logs show `No Account specified; resolving folder 'Invoices' - against the default profile store (robotuser1@test.com)` → `Could not find folder ...; stores available - to this profile: robotuser1@test.com, accounts-payable@test.com`. - -The smoking gun is that the shared `accounts-payable@test.com` store IS attached to the profile but was -never searched, because `Account` was blank — the fix is `Account`, not the folder string. - -## Sibling comparison (same playbook, Branch 1) - -| Sub-cause | Signature | This scenario? | -|---|---|---| -| Nested backslash path (`Inbox\Invoices`) | `folder does not exist`; resolved against default profile | No — `mail-get-outlook-mail-folder-not-found` | -| **Shared mailbox, `Account` blank** | **`folder does not exist`; folder is in a shared store the lookup never searched** | **Yes** | -| Localized folder name (e.g. `Posteingang`) | `folder does not exist`; name mismatches UI language | Not yet covered | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `GetOutlookMailMessages` with `Account=""`, `MailFolder="Invoices"`, no `Filter` | -| `data/m/r/` | synthetic canned `uip` responses; the smoking gun is in `job-logs.json` (No Account specified → resolving against default profile → folder not found there, shared store listed among available stores); `docsai ask` passthrough | - -Diagnosis is not leaked in any agent-visible name (project `PayablesInvoiceReader`, activity "Get -Outlook Mail Messages"). The prompt states observed symptoms only: the read fails saying the folder does -not exist, yet the folder is definitely present — in the Accounts Payable **shared** mailbox the user -has open alongside their own inbox. - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent attributed the failure to the shared-mailbox folder being read with a blank `Account` (default- - profile-only scope), and prescribed setting `Account` to the shared mailbox's SMTP address — not a - folder-string edit, COM/session, timeout, or filter fix. - -Playbook: `references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` -(Branch 1 — shared-mailbox / `Account` omission sub-cause). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/e2a5346a88.json deleted file mode 100644 index db7151ab83..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:10:00.190Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:10:01.020Z", - "Message": "[Get Outlook Mail Messages] Attached to running Outlook instance under UIPATH\\ROBOTUSER1" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-04T07:10:01.640Z", - "Message": "[Get Outlook Mail Messages] No Account specified; resolving folder 'Invoices' against the default profile store (robotuser1@test.com)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-04T07:10:02.190Z", - "Message": "[Get Outlook Mail Messages] Could not find folder 'Invoices' under the default profile store (robotuser1@test.com); stores available to this profile: robotuser1@test.com, accounts-payable@test.com" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T07:10:02.360Z", - "Message": "Get Outlook Mail Messages: The specified folder does not exist.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:10:02.400Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/manifest.json deleted file mode 100644 index f29f3401bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/m/r/manifest.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Get Outlook Mail Messages shared-mailbox-Account sub-case (Branch 1 of get-outlook-mail-failures); mock dispatcher only -- do not treat as investigation evidence. The MailFolder ('Invoices') is a plain top-level folder that exists in a SHARED mailbox, but Account is blank so the activity resolves it against the default profile only and raises 'The specified folder does not exist'. docsai proxied to the real CLI; the --level Error logs rule precedes the generic logs rule. Matching is token-aware (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs --level Error", - "file": "8f63dd2da5.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/uip-fixture.json new file mode 100644 index 0000000000..0593de73bd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T07:10:02.360Z\",\r\n \"Message\": \"Get Outlook Mail Messages: The specified folder does not exist.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-6f70-4a81-9b2c-3d4e5f607182\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-7081-4a92-9b3c-4d5e6f708293\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PayablesInvoiceReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T07:10:00.150Z\",\r\n \"EndTime\": \"2026-06-04T07:10:02.410Z\",\r\n \"CreationTime\": \"2026-06-04T07:09:59.900Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:10:00.190Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:10:01.020Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Attached to running Outlook instance under UIPATH\\\\ROBOTUSER1\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-04T07:10:01.640Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] No Account specified; resolving folder 'Invoices' against the default profile store (robotuser1@test.com)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-04T07:10:02.190Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] Could not find folder 'Invoices' under the default profile store (robotuser1@test.com); stores available to this profile: robotuser1@test.com, accounts-payable@test.com\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T07:10:02.360Z\",\r\n \"Message\": \"Get Outlook Mail Messages: The specified folder does not exist.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:10:02.400Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3d4e5f6-7081-4a92-9b3c-4d5e6f708293\",\r\n \"StartTime\": \"2026-06-04T07:10:00.150Z\",\r\n \"EndTime\": \"2026-06-04T07:10:02.410Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"d4e5f6a7-8192-4b03-9c4d-5e6f70819304\",\r\n \"Info\": \"Get Outlook Mail Messages: The specified folder does not exist.\\r\\n\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.\",\r\n \"CreationTime\": \"2026-06-04T07:09:59.900Z\",\r\n \"ReleaseName\": \"PayablesInvoiceReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6041407,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c3d4e5-6f70-4a81-9b2c-3d4e5f607182\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T07:10:02.410Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3d4e5f6-7081-4a92-9b3c-4d5e6f708293\",\r\n \"Id\": 682511884\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/task.yaml index 3a5d033dd3..db166774c5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-shared-mailbox-account/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/README.md new file mode 100644 index 0000000000..7cc1c75345 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/README.md @@ -0,0 +1,73 @@ +# Get Outlook Mail Messages Failure - Timeout on a Large Folder + +This scenario reproduces a runtime `Get Outlook Mail Messages` fault +caused by **message volume vs the timeout window**: the activity reads +the `Archive` folder (approx. 18,420 items) with `Top="5000"` and +`OnlyUnreadMessages="False"`, leaving the default `TimeoutMS="30000"`. +Enumerating that many messages cannot complete inside 30 seconds, so the +activity raises +`System.TimeoutException: The operation has timed out.` + +## What this scenario uncovers + +**Root Cause:** The `Get Outlook Mail Messages` activity in `Main.xaml` +asks for up to 5,000 messages from a folder of ~18,420 items with no +unread-only filter, against a 30-second `TimeoutMS`. The read cannot +finish in the window, so the job faults at the full 30s boundary. The +`MailFolder` string, the `Account`, and the Outlook/COM session are all +fine - this is a volume-vs-timeout problem. + +This maps to: +`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` +(Branch 2 - Timeout on a large folder). + +The correct fix raises `TimeoutMS` (e.g. 60000+) **and** reduces the batch +(`Top` ~10-20, enable `OnlyUnreadMessages`, or add a `Filter`); for a +chronically huge folder, migrate to the Graph o365-activities. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: `Get Outlook Mail Messages` on `Archive` with `Top="5000"`, `OnlyUnreadMessages="False"`, `TimeoutMS="30000"` | + +The smoking gun is in the full job logs +(`or-jobs-logs-b0c1d2e3-output-json.json`): an Info line that the +`Archive` folder holds `approx. 18,420 items; Top=5000, +OnlyUnreadMessages=False`, a Warn at +25s that enumeration was still +running as the 30s timeout approached, then the timeout Error - steering +the diagnosis toward volume vs `TimeoutMS` rather than a folder or COM +fault. + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Sibling-branch comparison + +| Branch | Signature | Why this scenario is NOT it | +|---|---|---| +| 1 - Folder not resolved | `The specified folder does not exist` | Error is a timeout; the folder enumerated and returned 4,100 items | +| **2 - Timeout on a large folder** | **`The operation has timed out` after `TimeoutMS`** | **This scenario** - ~18k items, `Top=5000`, default 30s | +| 3 - Outlook not running / COM / privilege | `Outlook is not running` / COM cast / freeze | Outlook responded and returned items; no COM/privilege error | +| 4 - Malformed `Filter` | `Filter` error or zero results with a filter set | No `Filter` is configured on the activity | +| 5 - Cached Exchange Mode desync | **No exception**, mail silently missing | Job hard-faults with a timeout, not a silent miss | + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `get-outlook-mail-failures.md` Branch 2 (timeout on a + large folder). +- Agent attributed the fault to message volume vs `TimeoutMS` (`Top` + high/unset, `OnlyUnreadMessages` off) and recommended raising + `TimeoutMS` **and** reducing the batch (`Top` / `OnlyUnreadMessages` / + `Filter`). + +This scenario **scores the conclusion, not the trajectory**: the +`llm_judge` grades the agent's final response and tool calls against +`RESOLUTION.md`, not any specific path through the investigation or any +internal `.local/investigations/` state. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/13629ea3ce.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/13629ea3ce.json deleted file mode 100644 index c42112f49e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/13629ea3ce.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/474c9df6a4.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/474c9df6a4.json deleted file mode 100644 index 320f78a65f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/474c9df6a4.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678", - "StartTime": "2026-06-03T02:00:00.180Z", - "EndTime": "2026-06-03T02:00:30.620Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "c1d2e3f4-5a6b-4c7d-8e90-1f2a3b4c5d6e", - "Info": "Get Outlook Mail Messages: The operation has timed out.\r\n\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out.", - "CreationTime": "2026-06-03T02:00:00.010Z", - "ReleaseName": "ArchiveMailScanner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6029417, - "OrganizationUnitId": 7791618, - "FolderKey": "a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-03T02:00:30.620Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678", - "Id": 682419703 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/5f509f7ef5.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/5f509f7ef5.json deleted file mode 100644 index d992583111..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/5f509f7ef5.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:00:00.210Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:00:00.512Z", - "Message": "[Get Outlook Mail Messages] enumerating folder 'Archive' (approx. 18,420 items); Top=5000, OnlyUnreadMessages=False" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-03T02:00:12.004Z", - "Message": "[Get Outlook Mail Messages] retrieved 4,100 of 5,000 requested items so far" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-03T02:00:25.733Z", - "Message": "[Get Outlook Mail Messages] enumeration has been running for more than 25s; folder is large and TimeoutMS=30000 is about to be reached" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:00:30.541Z", - "Message": "[Get Outlook Mail Messages] The operation has timed out." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:00:30.602Z", - "Message": "The operation has timed out.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:00:30.615Z", - "Message": "Execution ended in: 00:00:30" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/README.md deleted file mode 100644 index df1b3bf6f5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/README.md +++ /dev/null @@ -1,72 +0,0 @@ -# Get Outlook Mail Messages Failure - Timeout on a Large Folder - -This scenario reproduces a runtime `Get Outlook Mail Messages` fault -caused by **message volume vs the timeout window**: the activity reads -the `Archive` folder (approx. 18,420 items) with `Top="5000"` and -`OnlyUnreadMessages="False"`, leaving the default `TimeoutMS="30000"`. -Enumerating that many messages cannot complete inside 30 seconds, so the -activity raises -`System.TimeoutException: The operation has timed out.` - -## What this scenario uncovers - -**Root Cause:** The `Get Outlook Mail Messages` activity in `Main.xaml` -asks for up to 5,000 messages from a folder of ~18,420 items with no -unread-only filter, against a 30-second `TimeoutMS`. The read cannot -finish in the window, so the job faults at the full 30s boundary. The -`MailFolder` string, the `Account`, and the Outlook/COM session are all -fine - this is a volume-vs-timeout problem. - -This maps to: -`references/activity-packages/mail-activities/playbooks/get-outlook-mail-failures.md` -(Branch 2 - Timeout on a large folder). - -The correct fix raises `TimeoutMS` (e.g. 60000+) **and** reduces the batch -(`Top` ~10-20, enable `OnlyUnreadMessages`, or add a `Filter`); for a -chronically huge folder, migrate to the Graph o365-activities. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: `Get Outlook Mail Messages` on `Archive` with `Top="5000"`, `OnlyUnreadMessages="False"`, `TimeoutMS="30000"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The smoking gun is in the full job logs -(`or-jobs-logs-b0c1d2e3-output-json.json`): an Info line that the -`Archive` folder holds `approx. 18,420 items; Top=5000, -OnlyUnreadMessages=False`, a Warn at +25s that enumeration was still -running as the 30s timeout approached, then the timeout Error - steering -the diagnosis toward volume vs `TimeoutMS` rather than a folder or COM -fault. - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Sibling-branch comparison - -| Branch | Signature | Why this scenario is NOT it | -|---|---|---| -| 1 - Folder not resolved | `The specified folder does not exist` | Error is a timeout; the folder enumerated and returned 4,100 items | -| **2 - Timeout on a large folder** | **`The operation has timed out` after `TimeoutMS`** | **This scenario** - ~18k items, `Top=5000`, default 30s | -| 3 - Outlook not running / COM / privilege | `Outlook is not running` / COM cast / freeze | Outlook responded and returned items; no COM/privilege error | -| 4 - Malformed `Filter` | `Filter` error or zero results with a filter set | No `Filter` is configured on the activity | -| 5 - Cached Exchange Mode desync | **No exception**, mail silently missing | Job hard-faults with a timeout, not a silent miss | - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `get-outlook-mail-failures.md` Branch 2 (timeout on a - large folder). -- Agent attributed the fault to message volume vs `TimeoutMS` (`Top` - high/unset, `OnlyUnreadMessages` off) and recommended raising - `TimeoutMS` **and** reducing the batch (`Top` / `OnlyUnreadMessages` / - `Filter`). - -This scenario **scores the conclusion, not the trajectory**: the -`llm_judge` grades the agent's final response and tool calls against -`RESOLUTION.md`, not any specific path through the investigation or any -internal `.local/investigations/` state. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/a89a649ff3.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/a89a649ff3.json deleted file mode 100644 index a7ea8c2b7e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/a89a649ff3.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678", - "State": "Faulted", - "ProcessName": "ArchiveMailScanner", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T02:00:00.180Z", - "EndTime": "2026-06-03T02:00:30.620Z", - "CreationTime": "2026-06-03T02:00:00.010Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/ea9cb3ca0f.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/ea9cb3ca0f.json deleted file mode 100644 index 1f09f3f2c4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/ea9cb3ca0f.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:00:30.541Z", - "Message": "[Get Outlook Mail Messages] The operation has timed out." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:00:30.602Z", - "Message": "The operation has timed out.\r\nin Main.xaml\r\n at GetOutlookMailMessages \"Get Outlook Mail Messages\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/manifest.json deleted file mode 100644 index c2af47ed78..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Get Outlook Mail Messages scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "13629ea3ce.json" - }, - { - "match": "or jobs list --folder-key \"a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456\" --state Faulted --output json", - "file": "a89a649ff3.json" - }, - { - "match": "or jobs list --folder-key a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456 --state Faulted --output json", - "file": "a89a649ff3.json" - }, - { - "match": "or jobs get \"b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678\" --output json", - "file": "474c9df6a4.json" - }, - { - "match": "or jobs get b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678 --output json", - "file": "474c9df6a4.json" - }, - { - "match": "or jobs logs \"b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678\" --level Error --output json", - "file": "ea9cb3ca0f.json" - }, - { - "match": "or jobs logs b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678 --level Error --output json", - "file": "ea9cb3ca0f.json" - }, - { - "match": "or jobs logs \"b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678\" --output json", - "file": "5f509f7ef5.json" - }, - { - "match": "or jobs logs b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678 --output json", - "file": "5f509f7ef5.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/uip-fixture.json new file mode 100644 index 0000000000..e0a6d4d1ec --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ArchiveMailScanner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T02:00:00.180Z\",\r\n \"EndTime\": \"2026-06-03T02:00:30.620Z\",\r\n \"CreationTime\": \"2026-06-03T02:00:00.010Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:00:30.541Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] The operation has timed out.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:00:30.602Z\",\r\n \"Message\": \"The operation has timed out.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678\",\r\n \"StartTime\": \"2026-06-03T02:00:00.180Z\",\r\n \"EndTime\": \"2026-06-03T02:00:30.620Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"c1d2e3f4-5a6b-4c7d-8e90-1f2a3b4c5d6e\",\r\n \"Info\": \"Get Outlook Mail Messages: The operation has timed out.\\r\\n\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\",\r\n \"CreationTime\": \"2026-06-03T02:00:00.010Z\",\r\n \"ReleaseName\": \"ArchiveMailScanner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6029417,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-03T02:00:30.620Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678\",\r\n \"Id\": 682419703\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b0c1d2e3-4f5a-4b6c-9d7e-8f9012345678", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:00:00.210Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:00:00.512Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] enumerating folder 'Archive' (approx. 18,420 items); Top=5000, OnlyUnreadMessages=False\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-03T02:00:12.004Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] retrieved 4,100 of 5,000 requested items so far\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-03T02:00:25.733Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] enumeration has been running for more than 25s; folder is large and TimeoutMS=30000 is about to be reached\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:00:30.541Z\",\r\n \"Message\": \"[Get Outlook Mail Messages] The operation has timed out.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:00:30.602Z\",\r\n \"Message\": \"The operation has timed out.\\r\\nin Main.xaml\\r\\n at GetOutlookMailMessages \\\"Get Outlook Mail Messages\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:00:30.615Z\",\r\n \"Message\": \"Execution ended in: 00:00:30\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a9b0c1d2-3e4f-4a5b-8c6d-7e8f90123456\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/task.yaml index 8fc4739c63..b35a780cc8 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-get-outlook-mail-timeout-large-folder/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/README.md new file mode 100644 index 0000000000..b1cd7d3649 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/README.md @@ -0,0 +1,84 @@ +# Move Outlook Mail Message Failure - COM Session Loss (Branch 1) + +This scenario reproduces a `Move Outlook Mail Message` +(`MoveOutlookMessage`, `UiPath.Mail.Outlook.Activities`) failure that +faults 100% on an unattended robot with `The operation failed` +(`System.Runtime.InteropServices.COMException`, RPC server unavailable +`0x800706BA`). The robot has no live local Outlook COM session - Classic +Outlook is installed but is not running/synced for the unattended +Windows profile - so the destination folder tree cannot be indexed. + +## What this scenario uncovers + +**Root Cause:** The move activity is a desktop COM client. On the +unattended robot (`MOCK-HOST`, `UIPATH\ROBOTUSER1`) there is no running, +synced Outlook for the robot's Windows user, so the activity cannot bind +to an Outlook COM server and cannot index the destination folder tree. +The configured destination `MailFolder` `Inbox\Processed` is a **valid** +nested path - the path is not the bug. The COM bind failure surfaces as +`The operation failed`. + +This maps to **Branch 1 (COM session loss)** of: +`references/activity-packages/mail-activities/playbooks/move-outlook-mail-failures.md` + +The user is framed as **off-host**, so correct agent behavior is to +diagnose from Orchestrator + the project source and then hand a +host-side check list (keep Outlook running / Startup apps / Retry Scope +/ migrate to Graph o365 for unattended) without fabricating host +actions. + +## Sibling-branch comparison + +The move-outlook-mail-failures playbook has four cause-branches. This +scenario isolates Branch 1 and is constructed so the other three are +visibly ruled out: + +| Branch | Cause | Signature | Why NOT this scenario | +|---|---|---|---| +| **1 (this)** | COM session loss - no live Outlook on unattended robot | "The operation failed" / COM server unreachable, 100% on unattended | This IS the scenario | +| 2 | Destination folder path / `Account` mismatch | Deterministic "folder does not exist" - bare leaf name or shared-mailbox `Account` gap | Destination `Inbox\Processed` is a valid nested path; error is a COM-bind failure, not folder resolution | +| 3 | Modern-vs-classic type mismatch | `Cannot convert type 'String' to 'IResource' / 'Office365Message'` | Classic `MoveOutlookMessage` wired to a `MailMessage` var (correct type); no cast error | +| 4 | New Outlook removed the desktop COM API | "Broke completely after an Office/Windows update" flipped host to New Outlook | Classic Outlook IS installed - it just was not running; no post-update break | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: classic `MoveOutlookMessage` with a valid `Inbox\Processed` destination and a `MailMessage` input | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +The smoking-gun job-log line: + +``` +[Move Outlook Mail Message] could not bind to a running Outlook instance +for UIPATH\ROBOTUSER1 (no live COM session); the destination folder tree +could not be indexed +``` + +preceded by `Running unattended on MOCK-HOST as UIPATH\ROBOTUSER1 (no +interactive desktop session)`, steers toward COM session loss and away +from a folder-path typo or a registration/bitness error. + +## Success criteria + +This scenario **scores the conclusion, not the trajectory** - the +`llm_judge` grades the agent's final response and tool calls against +`RESOLUTION.md`, not its internal investigation state. + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched the move-outlook-mail-failures playbook / **Branch 1** + (COM session loss) and read the playbook under + `references/.../mail-activities/playbooks/`. +- Agent attributed `The operation failed` to no live/synced Outlook COM + session on the unattended robot (NOT a folder-path typo, type + mismatch, registration/bitness error, New Outlook, or SMTP issue). +- Agent handed a host-side check list (keep Outlook running / add to + Startup apps / Retry Scope / migrate to Graph o365 Move Email for + unattended) without fabricating host actions it could not run. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/5a4f5cf274.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/5a4f5cf274.json deleted file mode 100644 index 77d7a9ba08..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/5a4f5cf274.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f", - "State": "Faulted", - "ProcessName": "MailboxTriageMover", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T03:00:00.140Z", - "EndTime": "2026-06-04T03:00:05.260Z", - "CreationTime": "2026-06-04T02:59:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/9ca905b210.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/9ca905b210.json deleted file mode 100644 index 7c2586d517..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/9ca905b210.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T03:00:05.190Z", - "Message": "[Move Outlook Mail Message] The operation failed." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T03:00:05.231Z", - "Message": "The operation failed.\r\nin Main.xaml\r\n at MoveOutlookMessage \"Move Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The operation failed. (Exception from HRESULT: 0x800706BA (RPC_S_SERVER_UNAVAILABLE))" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/README.md deleted file mode 100644 index 5e58e949aa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/README.md +++ /dev/null @@ -1,83 +0,0 @@ -# Move Outlook Mail Message Failure - COM Session Loss (Branch 1) - -This scenario reproduces a `Move Outlook Mail Message` -(`MoveOutlookMessage`, `UiPath.Mail.Outlook.Activities`) failure that -faults 100% on an unattended robot with `The operation failed` -(`System.Runtime.InteropServices.COMException`, RPC server unavailable -`0x800706BA`). The robot has no live local Outlook COM session - Classic -Outlook is installed but is not running/synced for the unattended -Windows profile - so the destination folder tree cannot be indexed. - -## What this scenario uncovers - -**Root Cause:** The move activity is a desktop COM client. On the -unattended robot (`MOCK-HOST`, `UIPATH\ROBOTUSER1`) there is no running, -synced Outlook for the robot's Windows user, so the activity cannot bind -to an Outlook COM server and cannot index the destination folder tree. -The configured destination `MailFolder` `Inbox\Processed` is a **valid** -nested path - the path is not the bug. The COM bind failure surfaces as -`The operation failed`. - -This maps to **Branch 1 (COM session loss)** of: -`references/activity-packages/mail-activities/playbooks/move-outlook-mail-failures.md` - -The user is framed as **off-host**, so correct agent behavior is to -diagnose from Orchestrator + the project source and then hand a -host-side check list (keep Outlook running / Startup apps / Retry Scope -/ migrate to Graph o365 for unattended) without fabricating host -actions. - -## Sibling-branch comparison - -The move-outlook-mail-failures playbook has four cause-branches. This -scenario isolates Branch 1 and is constructed so the other three are -visibly ruled out: - -| Branch | Cause | Signature | Why NOT this scenario | -|---|---|---|---| -| **1 (this)** | COM session loss - no live Outlook on unattended robot | "The operation failed" / COM server unreachable, 100% on unattended | This IS the scenario | -| 2 | Destination folder path / `Account` mismatch | Deterministic "folder does not exist" - bare leaf name or shared-mailbox `Account` gap | Destination `Inbox\Processed` is a valid nested path; error is a COM-bind failure, not folder resolution | -| 3 | Modern-vs-classic type mismatch | `Cannot convert type 'String' to 'IResource' / 'Office365Message'` | Classic `MoveOutlookMessage` wired to a `MailMessage` var (correct type); no cast error | -| 4 | New Outlook removed the desktop COM API | "Broke completely after an Office/Windows update" flipped host to New Outlook | Classic Outlook IS installed - it just was not running; no post-update break | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: classic `MoveOutlookMessage` with a valid `Inbox\Processed` destination and a `MailMessage` input | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 1 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table (folders list; jobs list --state Faulted; jobs get; jobs logs --level Error; jobs logs; docsai passthrough) | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -The smoking-gun job-log line: - -``` -[Move Outlook Mail Message] could not bind to a running Outlook instance -for UIPATH\ROBOTUSER1 (no live COM session); the destination folder tree -could not be indexed -``` - -preceded by `Running unattended on MOCK-HOST as UIPATH\ROBOTUSER1 (no -interactive desktop session)`, steers toward COM session loss and away -from a folder-path typo or a registration/bitness error. - -## Success criteria - -This scenario **scores the conclusion, not the trajectory** - the -`llm_judge` grades the agent's final response and tool calls against -`RESOLUTION.md`, not its internal investigation state. - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched the move-outlook-mail-failures playbook / **Branch 1** - (COM session loss) and read the playbook under - `references/.../mail-activities/playbooks/`. -- Agent attributed `The operation failed` to no live/synced Outlook COM - session on the unattended robot (NOT a folder-path typo, type - mismatch, registration/bitness error, New Outlook, or SMTP issue). -- Agent handed a host-side check list (keep Outlook running / add to - Startup apps / Retry Scope / migrate to Graph o365 Move Email for - unattended) without fabricating host actions it could not run. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/a00f3c6005.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/a00f3c6005.json deleted file mode 100644 index 799a1b6751..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/a00f3c6005.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f", - "StartTime": "2026-06-04T03:00:00.140Z", - "EndTime": "2026-06-04T03:00:05.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "e0f1a2b3-c4d5-4e6f-9081-2a3b4c5d6e7f", - "Info": "Move Outlook Mail Message: The operation failed.\r\n\r\nin Main.xaml\r\n at MoveOutlookMessage \"Move Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The operation failed. (Exception from HRESULT: 0x800706BA (RPC_S_SERVER_UNAVAILABLE)) The Outlook COM server could not be reached.", - "CreationTime": "2026-06-04T02:59:59.910Z", - "ReleaseName": "MailboxTriageMover", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6041922, - "OrganizationUnitId": 7791618, - "FolderKey": "b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T03:00:05.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f", - "Id": 690412877 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/d2af24743c.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/d2af24743c.json deleted file mode 100644 index 20c6b0769f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/d2af24743c.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T03:00:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T03:00:00.420Z", - "Message": "Running unattended on MOCK-HOST as UIPATH\\ROBOTUSER1 (no interactive desktop session)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T03:00:01.005Z", - "Message": "[Move Outlook Mail Message] Configured destination MailFolder: 'Inbox\\Processed' (Account: default profile)" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-04T03:00:03.640Z", - "Message": "[Move Outlook Mail Message] could not bind to a running Outlook instance for UIPATH\\ROBOTUSER1 (no live COM session); the destination folder tree could not be indexed" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T03:00:05.190Z", - "Message": "[Move Outlook Mail Message] The operation failed." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T03:00:05.231Z", - "Message": "The operation failed.\r\nin Main.xaml\r\n at MoveOutlookMessage \"Move Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The operation failed. (Exception from HRESULT: 0x800706BA (RPC_S_SERVER_UNAVAILABLE))" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T03:00:05.255Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/ee6313f934.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/ee6313f934.json deleted file mode 100644 index bc27540ded..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/ee6313f934.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the mailbox triage RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "d0e1f2a3-b4c5-4d6e-8f70-1a2b3c4d5e6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/manifest.json deleted file mode 100644 index 0b2fe80963..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Move Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "ee6313f934.json" - }, - { - "match": "or jobs list --folder-key \"b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e\" --state Faulted --output json", - "file": "5a4f5cf274.json" - }, - { - "match": "or jobs list --folder-key b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e --state Faulted --output json", - "file": "5a4f5cf274.json" - }, - { - "match": "or jobs get \"c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f\" --output json", - "file": "a00f3c6005.json" - }, - { - "match": "or jobs get c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f --output json", - "file": "a00f3c6005.json" - }, - { - "match": "or jobs logs \"c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f\" --level Error --output json", - "file": "9ca905b210.json" - }, - { - "match": "or jobs logs c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f --level Error --output json", - "file": "9ca905b210.json" - }, - { - "match": "or jobs logs \"c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f\" --output json", - "file": "d2af24743c.json" - }, - { - "match": "or jobs logs c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f --output json", - "file": "d2af24743c.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/uip-fixture.json new file mode 100644 index 0000000000..5389ae1dc3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailboxTriageMover\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T03:00:00.140Z\",\r\n \"EndTime\": \"2026-06-04T03:00:05.260Z\",\r\n \"CreationTime\": \"2026-06-04T02:59:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T03:00:05.190Z\",\r\n \"Message\": \"[Move Outlook Mail Message] The operation failed.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T03:00:05.231Z\",\r\n \"Message\": \"The operation failed.\\r\\nin Main.xaml\\r\\n at MoveOutlookMessage \\\"Move Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The operation failed. (Exception from HRESULT: 0x800706BA (RPC_S_SERVER_UNAVAILABLE))\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f\",\r\n \"StartTime\": \"2026-06-04T03:00:00.140Z\",\r\n \"EndTime\": \"2026-06-04T03:00:05.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"e0f1a2b3-c4d5-4e6f-9081-2a3b4c5d6e7f\",\r\n \"Info\": \"Move Outlook Mail Message: The operation failed.\\r\\n\\r\\nin Main.xaml\\r\\n at MoveOutlookMessage \\\"Move Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The operation failed. (Exception from HRESULT: 0x800706BA (RPC_S_SERVER_UNAVAILABLE)) The Outlook COM server could not be reached.\",\r\n \"CreationTime\": \"2026-06-04T02:59:59.910Z\",\r\n \"ReleaseName\": \"MailboxTriageMover\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6041922,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T03:00:05.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f\",\r\n \"Id\": 690412877\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c8d9e0f1-2a3b-4c4d-9e5f-6a7b8c9d0e1f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T03:00:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T03:00:00.420Z\",\r\n \"Message\": \"Running unattended on MOCK-HOST as UIPATH\\\\ROBOTUSER1 (no interactive desktop session)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T03:00:01.005Z\",\r\n \"Message\": \"[Move Outlook Mail Message] Configured destination MailFolder: 'Inbox\\\\Processed' (Account: default profile)\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-04T03:00:03.640Z\",\r\n \"Message\": \"[Move Outlook Mail Message] could not bind to a running Outlook instance for UIPATH\\\\ROBOTUSER1 (no live COM session); the destination folder tree could not be indexed\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T03:00:05.190Z\",\r\n \"Message\": \"[Move Outlook Mail Message] The operation failed.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T03:00:05.231Z\",\r\n \"Message\": \"The operation failed.\\r\\nin Main.xaml\\r\\n at MoveOutlookMessage \\\"Move Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The operation failed. (Exception from HRESULT: 0x800706BA (RPC_S_SERVER_UNAVAILABLE))\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T03:00:05.255Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b7c8d9e0-1f2a-4b3c-8d4e-5f6a7b8c9d0e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the mailbox triage RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"d0e1f2a3-b4c5-4d6e-8f70-1a2b3c4d5e6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/task.yaml index e664d08c7e..f92f4a8bcc 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-com-session-loss/task.yaml @@ -27,22 +27,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/README.md new file mode 100644 index 0000000000..c765716835 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/README.md @@ -0,0 +1,80 @@ +# Move Outlook Mail Message Failure - Bare Nested-Folder Leaf Name (Branch 2) + +This scenario reproduces a runtime `Move Outlook Mail Message` +(`MoveOutlookMessage`, `UiPath.Mail.Outlook.Activities`) failure that +faults **deterministically, on every run**, with +`The specified folder does not exist.` The destination `MailFolder` is set +to the bare leaf name `"Processed"` instead of the nested path +`Inbox\Processed`, so Outlook resolves it against the mailbox root, finds +no top-level folder by that name, and throws. The Outlook desktop COM +session is fine -- it attaches and resolves the source message before the +destination folder fails. + +## What this scenario uncovers + +**Root Cause:** The `Move Outlook Mail Message` activity in `Main.xaml` +has `MailFolder="Processed"`. `Processed` actually lives under `Inbox`, so +the bare leaf name does not resolve. The fix is to address the destination +by path -- `MailFolder="Inbox\Processed"` (or `Inbox/Processed`); for a +shared-mailbox destination, also set the activity's `Account` field. + +This maps to: +`references/activity-packages/mail-activities/playbooks/move-outlook-mail-failures.md` +(Branch 2 -- destination folder path / `Account`). + +This is a **workflow-fixable configuration bug** -- the agent can diagnose +and fix it entirely from the project source and Orchestrator evidence. No +host-side action is required, so the prompt is a plain "investigate why it +faults" with no off-host / host-command framing. + +## How this scenario reproduces Branch 2 + +The Branch 2 hallmark is "fails immediately and deterministically, every +run, even attended." This scenario encodes exactly that: + +| Branch 2 signal | Where it appears in this scenario | +|---|---| +| `The specified folder does not exist` | `jobs get` `Info` + the `jobs logs` error line | +| Bare nested-folder leaf name | `Main.xaml` `MailFolder="Processed"` | +| Deterministic, every run | job faulted ~2s after start; user reports "every run" | +| COM session is healthy (not Branch 1) | job log: Outlook attached + source message resolved in Inbox | +| Smoking-gun resolution line | `destination folder 'Processed' not found at the mailbox root; no such top-level folder` | + +## Sibling-branch comparison (why this is Branch 2, not 1/3/4) + +| Branch | Signature | Timing | This scenario? | +|---|---|---|---| +| 1 - COM session loss | "operation failed" / "folder does not exist" | **intermittent** or unattended-only | No -- COM attached fine; deterministic | +| **2 - Folder path / Account** | **"The specified folder does not exist"** | **immediate, every run** | **Yes** | +| 3 - Modern-vs-classic type mismatch | `Cannot convert type 'String' to 'IResource' / 'Office365Message'` | design-time / after dependency update | No -- no cast error; classic activity | +| 4 - New Outlook | COM-bind failure, "Outlook is not running" | broke completely after an OS/Office update | No -- no post-update break | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `MailFolder="Processed"` on the Move activity | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This test **scores the conclusion, not the trajectory.** The `llm_judge` +grades the agent's final response and tool calls against `RESOLUTION.md`; +it does not require a specific investigation path or any +`.local/investigations/` internal state. + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `move-outlook-mail-failures.md` Branch 2 (named the + playbook and Read a file under + `references/activity-packages/mail-activities/playbooks/`). +- Agent reached the same conclusion as `RESOLUTION.md`: the bare leaf + `MailFolder="Processed"` does not resolve (the folder lives under Inbox), + the Outlook COM session is fine, and the fix is the nested path + `Inbox\Processed` (or `Account` for a shared mailbox). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/2ed5559edf.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/2ed5559edf.json deleted file mode 100644 index 9c4043f497..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/2ed5559edf.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b", - "StartTime": "2026-06-04T07:20:00.140Z", - "EndTime": "2026-06-04T07:20:02.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "f1a2b3c4-5d6e-4f70-8192-a3b4c5d6e7f8", - "Info": "Move Outlook Mail Message: The specified folder does not exist.\r\n\r\nin Main.xaml\r\n at MoveOutlookMessage \"Move Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.", - "CreationTime": "2026-06-04T07:19:59.880Z", - "ReleaseName": "OrderMailFiler", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 7042918, - "OrganizationUnitId": 8810427, - "FolderKey": "d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T07:20:02.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b", - "Id": 703918264 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/33c5307b4b.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/33c5307b4b.json deleted file mode 100644 index dbc017bfe6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/33c5307b4b.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:20:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:20:00.642Z", - "Message": "[Move Outlook Mail Message] Attached to the running Outlook desktop session for profile UIPATH\\ROBOTUSER1" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:20:01.118Z", - "Message": "[Move Outlook Mail Message] Source message resolved (Subject: 'Order #48217 confirmation') in Inbox" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:20:01.604Z", - "Message": "[Move Outlook Mail Message] Resolving destination folder 'Processed' against the default profile root" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T07:20:02.205Z", - "Message": "[Move Outlook Mail Message] destination folder 'Processed' not found at the mailbox root; no such top-level folder. The specified folder does not exist." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T07:20:02.241Z", - "Message": "Move Outlook Mail Message: The specified folder does not exist.\r\nin Main.xaml\r\n at MoveOutlookMessage \"Move Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T07:20:02.255Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/6814c2ec5c.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/6814c2ec5c.json deleted file mode 100644 index 1b8396265b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/6814c2ec5c.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the order-processing RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-5a6b-4c7d-9e0f-1a2b3c4d5e6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/README.md deleted file mode 100644 index 554d7d7ca2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/README.md +++ /dev/null @@ -1,79 +0,0 @@ -# Move Outlook Mail Message Failure - Bare Nested-Folder Leaf Name (Branch 2) - -This scenario reproduces a runtime `Move Outlook Mail Message` -(`MoveOutlookMessage`, `UiPath.Mail.Outlook.Activities`) failure that -faults **deterministically, on every run**, with -`The specified folder does not exist.` The destination `MailFolder` is set -to the bare leaf name `"Processed"` instead of the nested path -`Inbox\Processed`, so Outlook resolves it against the mailbox root, finds -no top-level folder by that name, and throws. The Outlook desktop COM -session is fine -- it attaches and resolves the source message before the -destination folder fails. - -## What this scenario uncovers - -**Root Cause:** The `Move Outlook Mail Message` activity in `Main.xaml` -has `MailFolder="Processed"`. `Processed` actually lives under `Inbox`, so -the bare leaf name does not resolve. The fix is to address the destination -by path -- `MailFolder="Inbox\Processed"` (or `Inbox/Processed`); for a -shared-mailbox destination, also set the activity's `Account` field. - -This maps to: -`references/activity-packages/mail-activities/playbooks/move-outlook-mail-failures.md` -(Branch 2 -- destination folder path / `Account`). - -This is a **workflow-fixable configuration bug** -- the agent can diagnose -and fix it entirely from the project source and Orchestrator evidence. No -host-side action is required, so the prompt is a plain "investigate why it -faults" with no off-host / host-command framing. - -## How this scenario reproduces Branch 2 - -The Branch 2 hallmark is "fails immediately and deterministically, every -run, even attended." This scenario encodes exactly that: - -| Branch 2 signal | Where it appears in this scenario | -|---|---| -| `The specified folder does not exist` | `jobs get` `Info` + the `jobs logs` error line | -| Bare nested-folder leaf name | `Main.xaml` `MailFolder="Processed"` | -| Deterministic, every run | job faulted ~2s after start; user reports "every run" | -| COM session is healthy (not Branch 1) | job log: Outlook attached + source message resolved in Inbox | -| Smoking-gun resolution line | `destination folder 'Processed' not found at the mailbox root; no such top-level folder` | - -## Sibling-branch comparison (why this is Branch 2, not 1/3/4) - -| Branch | Signature | Timing | This scenario? | -|---|---|---|---| -| 1 - COM session loss | "operation failed" / "folder does not exist" | **intermittent** or unattended-only | No -- COM attached fine; deterministic | -| **2 - Folder path / Account** | **"The specified folder does not exist"** | **immediate, every run** | **Yes** | -| 3 - Modern-vs-classic type mismatch | `Cannot convert type 'String' to 'IResource' / 'Office365Message'` | design-time / after dependency update | No -- no cast error; classic activity | -| 4 - New Outlook | COM-bind failure, "Outlook is not running" | broke completely after an OS/Office update | No -- no post-update break | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `MailFolder="Processed"` on the Move activity | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table (quoted + unquoted arg variants) | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This test **scores the conclusion, not the trajectory.** The `llm_judge` -grades the agent's final response and tool calls against `RESOLUTION.md`; -it does not require a specific investigation path or any -`.local/investigations/` internal state. - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `move-outlook-mail-failures.md` Branch 2 (named the - playbook and Read a file under - `references/activity-packages/mail-activities/playbooks/`). -- Agent reached the same conclusion as `RESOLUTION.md`: the bare leaf - `MailFolder="Processed"` does not resolve (the folder lives under Inbox), - the Outlook COM session is fine, and the fix is the nested path - `Inbox\Processed` (or `Account` for a shared mailbox). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/dc2dbfc263.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/dc2dbfc263.json deleted file mode 100644 index 23a56a0599..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/dc2dbfc263.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T07:20:02.205Z", - "Message": "[Move Outlook Mail Message] destination folder 'Processed' not found at the mailbox root; no such top-level folder. The specified folder does not exist." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T07:20:02.241Z", - "Message": "Move Outlook Mail Message: The specified folder does not exist.\r\nin Main.xaml\r\n at MoveOutlookMessage \"Move Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/f164ffd674.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/f164ffd674.json deleted file mode 100644 index 9a01f650cc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/f164ffd674.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b", - "State": "Faulted", - "ProcessName": "OrderMailFiler", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T07:20:00.140Z", - "EndTime": "2026-06-04T07:20:02.260Z", - "CreationTime": "2026-06-04T07:19:59.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/manifest.json deleted file mode 100644 index a9a2f4ab5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Move Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6814c2ec5c.json" - }, - { - "match": "or jobs list --folder-key \"d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a\" --state Faulted --output json", - "file": "f164ffd674.json" - }, - { - "match": "or jobs list --folder-key d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a --state Faulted --output json", - "file": "f164ffd674.json" - }, - { - "match": "or jobs get \"e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b\" --output json", - "file": "2ed5559edf.json" - }, - { - "match": "or jobs get e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b --output json", - "file": "2ed5559edf.json" - }, - { - "match": "or jobs logs \"e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b\" --level Error --output json", - "file": "dc2dbfc263.json" - }, - { - "match": "or jobs logs e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b --level Error --output json", - "file": "dc2dbfc263.json" - }, - { - "match": "or jobs logs \"e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b\" --output json", - "file": "33c5307b4b.json" - }, - { - "match": "or jobs logs e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b --output json", - "file": "33c5307b4b.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/uip-fixture.json new file mode 100644 index 0000000000..5d96937410 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderMailFiler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T07:20:00.140Z\",\r\n \"EndTime\": \"2026-06-04T07:20:02.260Z\",\r\n \"CreationTime\": \"2026-06-04T07:19:59.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T07:20:02.205Z\",\r\n \"Message\": \"[Move Outlook Mail Message] destination folder 'Processed' not found at the mailbox root; no such top-level folder. The specified folder does not exist.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T07:20:02.241Z\",\r\n \"Message\": \"Move Outlook Mail Message: The specified folder does not exist.\\r\\nin Main.xaml\\r\\n at MoveOutlookMessage \\\"Move Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b\",\r\n \"StartTime\": \"2026-06-04T07:20:00.140Z\",\r\n \"EndTime\": \"2026-06-04T07:20:02.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"f1a2b3c4-5d6e-4f70-8192-a3b4c5d6e7f8\",\r\n \"Info\": \"Move Outlook Mail Message: The specified folder does not exist.\\r\\n\\r\\nin Main.xaml\\r\\n at MoveOutlookMessage \\\"Move Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Mail.Exception: The specified folder does not exist. ---> System.Exception: The specified folder does not exist.\",\r\n \"CreationTime\": \"2026-06-04T07:19:59.880Z\",\r\n \"ReleaseName\": \"OrderMailFiler\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 7042918,\r\n \"OrganizationUnitId\": 8810427,\r\n \"FolderKey\": \"d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T07:20:02.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b\",\r\n \"Id\": 703918264\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e0f1a2b3-4c5d-4e6f-9a7b-8c9d0e1f2a3b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:20:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:20:00.642Z\",\r\n \"Message\": \"[Move Outlook Mail Message] Attached to the running Outlook desktop session for profile UIPATH\\\\ROBOTUSER1\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:20:01.118Z\",\r\n \"Message\": \"[Move Outlook Mail Message] Source message resolved (Subject: 'Order #48217 confirmation') in Inbox\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:20:01.604Z\",\r\n \"Message\": \"[Move Outlook Mail Message] Resolving destination folder 'Processed' against the default profile root\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T07:20:02.205Z\",\r\n \"Message\": \"[Move Outlook Mail Message] destination folder 'Processed' not found at the mailbox root; no such top-level folder. The specified folder does not exist.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T07:20:02.241Z\",\r\n \"Message\": \"Move Outlook Mail Message: The specified folder does not exist.\\r\\nin Main.xaml\\r\\n at MoveOutlookMessage \\\"Move Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T07:20:02.255Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d9e0f1a2-3b4c-4d5e-8f6a-7b8c9d0e1f2a\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the order-processing RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-5a6b-4c7d-9e0f-1a2b3c4d5e6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/task.yaml index f64d36514b..4ad8fdf454 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-folder-path/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/README.md new file mode 100644 index 0000000000..9b7c74f26b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/README.md @@ -0,0 +1,76 @@ +# Move Outlook Mail Failure - Modern-vs-Classic Type Mismatch (Branch 3) + +This scenario reproduces a Move Outlook Mail Message failure caused by +a **modern-vs-classic type mismatch**. After a mail-package update the +`InboxAutoFiler` workflow uses the **modern Move Email** activity +(`MoveEmailConnections`, from `UiPath.MicrosoftOffice365.Activities`), +whose message input expects an `Office365Message` / `IResource` object +reference -- but a **`String`** message-ID variable (`messageId`) is +wired into that input, left over from the classic design. The job +faults with +`Cannot convert type 'System.String' to +'UiPath.MicrosoftOffice365.Models.Office365Message'` +(`System.InvalidCastException`). + +## What this scenario uncovers + +**Root Cause:** A `String` is bound to a modern Move Email property +that expects an `Office365Message` / `IResource` object. The cause is +the classic-vs-modern mix introduced when the team updated the mail +packages -- `project.json` now carries **both** +`UiPath.Mail.Activities` (classic) and +`UiPath.MicrosoftOffice365.Activities` (modern), and a classic string +binding was carried into the modern resource-typed property. It is the +**TYPE** that is wrong, not the message-ID value. + +This maps to: +`references/activity-packages/mail-activities/playbooks/move-outlook-mail-failures.md` +-- **Branch 3 (modern-vs-classic type mismatch)**. + +**Fix:** Map the message (and folder) with the **`+` (plus)** resource +picker so the binding resolves to the expected `Office365Message` / +`IResource` object, OR switch to the **classic** +`Move Outlook Mail Message` (`MoveOutlookMessage`) activity, which +accepts string / `MailMessage` inputs directly. + +The user is framed plainly ("investigate why it faults") -- this is a +workflow-fixable design/typing bug, so there is no off-host / +remote-user framing. + +## Sibling-branch comparison + +| Branch | Signature | Cause | This scenario? | +|---|---|---|---| +| 1 -- COM session loss | "operation failed" / "folder does not exist", intermittent or unattended-only | Outlook not running/synced for the Robot user | No | +| 2 -- Folder path / `Account` | "folder does not exist", immediate, every run | Bare nested leaf name, or shared-mailbox destination with empty `Account` | No | +| **3 -- Type mismatch** | **`Cannot convert type 'String' to '...Office365Message' / 'IResource'`** | **`String` wired into a modern resource-typed input after a package update** | **Yes** | +| 4 -- New Outlook | COM-bind / "Outlook is not running", broke completely after an OS/Office update | Host flipped to New Outlook; desktop COM API removed | No | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: modern Move Email activity with a `String` message-ID variable on its `Message` input; `project.json` carries both classic and modern mail packages | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `move-outlook-mail-failures.md` **Branch 3** and reached + the same conclusion as `RESOLUTION.md`: a `String` was wired into the + modern Move Email's resource-typed message input (classic/modern mix + after the package update), throwing the `String`-to-`Office365Message` + cast error -- and recommended the `+` resource picker OR the classic + `Move Outlook Mail Message` activity with string / `MailMessage` + inputs. + +This scenario **scores the conclusion, not the trajectory** -- the +`llm_judge` grades the agent's final response and tool calls against +`RESOLUTION.md`, never internal investigation state. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/6705551956.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/6705551956.json deleted file mode 100644 index 131aba681b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/6705551956.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the mail automation portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c4d5e6f7-8a9b-4c0d-9e1f-2a3b4c5d6e7f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/8bddc6e50a.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/8bddc6e50a.json deleted file mode 100644 index be49127b51..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/8bddc6e50a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d", - "StartTime": "2026-06-04T09:10:00.140Z", - "EndTime": "2026-06-04T09:10:02.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "b3c4d5e6-7f8a-4b9c-8d0e-1f2a3b4c5d6e", - "Info": "Move Email: Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\r\n\r\nin Main.xaml\r\n at MoveEmailConnections \"Move Email\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidCastException: Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.", - "CreationTime": "2026-06-04T09:09:59.910Z", - "ReleaseName": "InboxAutoFiler", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6041207, - "OrganizationUnitId": 7791618, - "FolderKey": "f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-04T09:10:02.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d", - "Id": 682340711 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/README.md deleted file mode 100644 index f929cbfe3f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/README.md +++ /dev/null @@ -1,75 +0,0 @@ -# Move Outlook Mail Failure - Modern-vs-Classic Type Mismatch (Branch 3) - -This scenario reproduces a Move Outlook Mail Message failure caused by -a **modern-vs-classic type mismatch**. After a mail-package update the -`InboxAutoFiler` workflow uses the **modern Move Email** activity -(`MoveEmailConnections`, from `UiPath.MicrosoftOffice365.Activities`), -whose message input expects an `Office365Message` / `IResource` object -reference -- but a **`String`** message-ID variable (`messageId`) is -wired into that input, left over from the classic design. The job -faults with -`Cannot convert type 'System.String' to -'UiPath.MicrosoftOffice365.Models.Office365Message'` -(`System.InvalidCastException`). - -## What this scenario uncovers - -**Root Cause:** A `String` is bound to a modern Move Email property -that expects an `Office365Message` / `IResource` object. The cause is -the classic-vs-modern mix introduced when the team updated the mail -packages -- `project.json` now carries **both** -`UiPath.Mail.Activities` (classic) and -`UiPath.MicrosoftOffice365.Activities` (modern), and a classic string -binding was carried into the modern resource-typed property. It is the -**TYPE** that is wrong, not the message-ID value. - -This maps to: -`references/activity-packages/mail-activities/playbooks/move-outlook-mail-failures.md` --- **Branch 3 (modern-vs-classic type mismatch)**. - -**Fix:** Map the message (and folder) with the **`+` (plus)** resource -picker so the binding resolves to the expected `Office365Message` / -`IResource` object, OR switch to the **classic** -`Move Outlook Mail Message` (`MoveOutlookMessage`) activity, which -accepts string / `MailMessage` inputs directly. - -The user is framed plainly ("investigate why it faults") -- this is a -workflow-fixable design/typing bug, so there is no off-host / -remote-user framing. - -## Sibling-branch comparison - -| Branch | Signature | Cause | This scenario? | -|---|---|---|---| -| 1 -- COM session loss | "operation failed" / "folder does not exist", intermittent or unattended-only | Outlook not running/synced for the Robot user | No | -| 2 -- Folder path / `Account` | "folder does not exist", immediate, every run | Bare nested leaf name, or shared-mailbox destination with empty `Account` | No | -| **3 -- Type mismatch** | **`Cannot convert type 'String' to '...Office365Message' / 'IResource'`** | **`String` wired into a modern resource-typed input after a package update** | **Yes** | -| 4 -- New Outlook | COM-bind / "Outlook is not running", broke completely after an OS/Office update | Host flipped to New Outlook; desktop COM API removed | No | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: modern Move Email activity with a `String` message-ID variable on its `Message` input; `project.json` carries both classic and modern mail packages | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 3 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `move-outlook-mail-failures.md` **Branch 3** and reached - the same conclusion as `RESOLUTION.md`: a `String` was wired into the - modern Move Email's resource-typed message input (classic/modern mix - after the package update), throwing the `String`-to-`Office365Message` - cast error -- and recommended the `+` resource picker OR the classic - `Move Outlook Mail Message` activity with string / `MailMessage` - inputs. - -This scenario **scores the conclusion, not the trajectory** -- the -`llm_judge` grades the agent's final response and tool calls against -`RESOLUTION.md`, never internal investigation state. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/a874f679f1.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/a874f679f1.json deleted file mode 100644 index eafb3b803c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/a874f679f1.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-04T09:10:02.190Z", - "Message": "[Move Email] Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T09:10:02.241Z", - "Message": "Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\r\nin Main.xaml\r\n at MoveEmailConnections \"Move Email\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/b3b28f95e5.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/b3b28f95e5.json deleted file mode 100644 index 9c1d2c5087..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/b3b28f95e5.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d", - "State": "Faulted", - "ProcessName": "InboxAutoFiler", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-04T09:10:00.140Z", - "EndTime": "2026-06-04T09:10:02.260Z", - "CreationTime": "2026-06-04T09:09:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/cb764210eb.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/cb764210eb.json deleted file mode 100644 index 3992bf3184..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/cb764210eb.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-04T09:10:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T09:10:01.402Z", - "Message": "[Move Email] Activity started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T09:10:01.928Z", - "Message": "[Move Email] message input bound to String 'AAMkAGI2aaaa0000bbbb1111cccc2222dddd3333eeee4444ffff5555AAA=' but the activity expects an Office365Message resource" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T09:10:02.190Z", - "Message": "[Move Email] Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-04T09:10:02.241Z", - "Message": "Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\r\nin Main.xaml\r\n at MoveEmailConnections \"Move Email\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-04T09:10:02.255Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/manifest.json deleted file mode 100644 index 6d01591e27..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Move Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6705551956.json" - }, - { - "match": "or jobs list --folder-key \"f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c\" --state Faulted --output json", - "file": "b3b28f95e5.json" - }, - { - "match": "or jobs list --folder-key f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c --state Faulted --output json", - "file": "b3b28f95e5.json" - }, - { - "match": "or jobs get \"a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d\" --output json", - "file": "8bddc6e50a.json" - }, - { - "match": "or jobs get a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d --output json", - "file": "8bddc6e50a.json" - }, - { - "match": "or jobs logs \"a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d\" --level Error --output json", - "file": "a874f679f1.json" - }, - { - "match": "or jobs logs a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d --level Error --output json", - "file": "a874f679f1.json" - }, - { - "match": "or jobs logs \"a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d\" --output json", - "file": "cb764210eb.json" - }, - { - "match": "or jobs logs a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d --output json", - "file": "cb764210eb.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..301c2a9e27 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxAutoFiler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-04T09:10:00.140Z\",\r\n \"EndTime\": \"2026-06-04T09:10:02.260Z\",\r\n \"CreationTime\": \"2026-06-04T09:09:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T09:10:02.190Z\",\r\n \"Message\": \"[Move Email] Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T09:10:02.241Z\",\r\n \"Message\": \"Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\\r\\nin Main.xaml\\r\\n at MoveEmailConnections \\\"Move Email\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d\",\r\n \"StartTime\": \"2026-06-04T09:10:00.140Z\",\r\n \"EndTime\": \"2026-06-04T09:10:02.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"b3c4d5e6-7f8a-4b9c-8d0e-1f2a3b4c5d6e\",\r\n \"Info\": \"Move Email: Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\\r\\n\\r\\nin Main.xaml\\r\\n at MoveEmailConnections \\\"Move Email\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidCastException: Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\",\r\n \"CreationTime\": \"2026-06-04T09:09:59.910Z\",\r\n \"ReleaseName\": \"InboxAutoFiler\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6041207,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-04T09:10:02.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d\",\r\n \"Id\": 682340711\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a2b3c4d5-6e7f-4a8b-9c0d-0e1f2a3b4c5d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T09:10:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T09:10:01.402Z\",\r\n \"Message\": \"[Move Email] Activity started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T09:10:01.928Z\",\r\n \"Message\": \"[Move Email] message input bound to String 'AAMkAGI2aaaa0000bbbb1111cccc2222dddd3333eeee4444ffff5555AAA=' but the activity expects an Office365Message resource\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T09:10:02.190Z\",\r\n \"Message\": \"[Move Email] Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-04T09:10:02.241Z\",\r\n \"Message\": \"Cannot convert type 'System.String' to 'UiPath.MicrosoftOffice365.Models.Office365Message'.\\r\\nin Main.xaml\\r\\n at MoveEmailConnections \\\"Move Email\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-04T09:10:02.255Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f1a2b3c4-5d6e-4f7a-8b9c-9d0e1f2a3b4c\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the mail automation portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c4d5e6f7-8a9b-4c0d-9e1f-2a3b4c5d6e7f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/task.yaml index bd1e54832a..57ffcf5f82 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-move-outlook-mail-type-mismatch/task.yaml @@ -28,22 +28,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/README.md new file mode 100644 index 0000000000..ba02524a81 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/README.md @@ -0,0 +1,73 @@ +# Send Outlook Mail Message Failure - COM Cast / Library Not Registered (Branch 1) + +This scenario reproduces a runtime `Send Outlook Mail Message` +(`SendOutlookMail`, `UiPath.Mail.Outlook.Activities`) failure at the +**COM layer**: the activity cannot bind the Outlook COM server and +faults with +`System.InvalidCastException: Unable to cast COM object of type +'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type +'Microsoft.Office.Interop.Outlook._Application'` and an inner +`COMException: Library not registered. (Exception from HRESULT: +0x8002801D (TYPE_E_LIBNOTREGISTERED))`. + +## What this scenario uncovers + +**Root Cause:** A process-vs-Outlook **bitness mismatch** (the Robot +executor process is 64-bit - `project.json` `targetFramework: "Windows"` +- against a 32-bit desktop Outlook on MOCK-HOST), or a **corrupted +Office registration / type library**. The `Send Outlook Mail Message` +activity drives the local Outlook desktop application through COM; the +bind fails before any message is composed. The `To` / `Subject` / +`Body` inputs are valid literals and are not the cause. + +This maps to: +`references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` +(Branch 1 - COM cast / library not registered). + +The user is framed as **off-host** (Orchestrator access only), so the +correct agent behavior is to diagnose from Orchestrator and hand the +user a **host-side check list** (matching-bitness desktop Outlook, +Office Quick Repair, clear orphaned `OUTLOOK.EXE`, optionally migrate +to Send SMTP Mail Message / Microsoft Graph o365 activities) rather +than claim to have run host commands itself. + +## Sibling-branch comparison + +The Send Outlook Mail Message playbook has three cause-branches. This +scenario exercises Branch 1; the judge rejects the other two. + +| Branch | Signature | Cause | This scenario | +|---|---|---|---| +| **1 (this)** | `InvalidCastException` to `_Application` / `COMException: Library not registered` (0x8002801D `TYPE_E_LIBNOTREGISTERED`, `REGDB_E_CLASSNOTREG`) | Outlook not installed / bitness mismatch / corrupted Office registration / orphaned `OUTLOOK.EXE` on the host | **Yes** - 64-bit process vs 32-bit Outlook; host-side fix | +| 2 | Activity times out (`TimeoutMS` elapsed) or job hangs with no exception | Hidden Outlook security prompt, Work Offline, or slow profile load | No - rejected (no timeout/hang; fails fast with a cast) | +| 3 | `NullReferenceException` at the activity | Uninitialized `To`/`Subject`/`Body` or empty attachment path | No - rejected (inputs are valid literals) | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: a `SendOutlookMail` activity with valid literal inputs, `targetFramework "Windows"` (64-bit), dependency `UiPath.Mail.Activities [1.18.3]` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This test **scores the conclusion, not the trajectory.** The +`llm_judge` grades the agent's final response and tool calls against +`RESOLUTION.md`; it does not require any specific investigation path or +internal-state file. + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `send-outlook-mail-failures.md` Branch 1 (COM cast / + library not registered). +- Agent attributed the failure to the Outlook COM registration / + bitness mismatch on the host and handed the user a host-side check + list (matching-bitness Outlook, Office Quick Repair, clear orphaned + `OUTLOOK.EXE`; optionally SMTP/Graph fallback) without fabricating + host actions it could not run. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/406f81c8d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/406f81c8d3.json deleted file mode 100644 index b4010569fc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/406f81c8d3.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-02T06:15:03.401Z", - "Message": "[Send Outlook Mail Message] Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. (Library not registered. Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T06:15:03.452Z", - "Message": "Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/README.md deleted file mode 100644 index fb5595829e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/README.md +++ /dev/null @@ -1,72 +0,0 @@ -# Send Outlook Mail Message Failure - COM Cast / Library Not Registered (Branch 1) - -This scenario reproduces a runtime `Send Outlook Mail Message` -(`SendOutlookMail`, `UiPath.Mail.Outlook.Activities`) failure at the -**COM layer**: the activity cannot bind the Outlook COM server and -faults with -`System.InvalidCastException: Unable to cast COM object of type -'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type -'Microsoft.Office.Interop.Outlook._Application'` and an inner -`COMException: Library not registered. (Exception from HRESULT: -0x8002801D (TYPE_E_LIBNOTREGISTERED))`. - -## What this scenario uncovers - -**Root Cause:** A process-vs-Outlook **bitness mismatch** (the Robot -executor process is 64-bit - `project.json` `targetFramework: "Windows"` -- against a 32-bit desktop Outlook on MOCK-HOST), or a **corrupted -Office registration / type library**. The `Send Outlook Mail Message` -activity drives the local Outlook desktop application through COM; the -bind fails before any message is composed. The `To` / `Subject` / -`Body` inputs are valid literals and are not the cause. - -This maps to: -`references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` -(Branch 1 - COM cast / library not registered). - -The user is framed as **off-host** (Orchestrator access only), so the -correct agent behavior is to diagnose from Orchestrator and hand the -user a **host-side check list** (matching-bitness desktop Outlook, -Office Quick Repair, clear orphaned `OUTLOOK.EXE`, optionally migrate -to Send SMTP Mail Message / Microsoft Graph o365 activities) rather -than claim to have run host commands itself. - -## Sibling-branch comparison - -The Send Outlook Mail Message playbook has three cause-branches. This -scenario exercises Branch 1; the judge rejects the other two. - -| Branch | Signature | Cause | This scenario | -|---|---|---|---| -| **1 (this)** | `InvalidCastException` to `_Application` / `COMException: Library not registered` (0x8002801D `TYPE_E_LIBNOTREGISTERED`, `REGDB_E_CLASSNOTREG`) | Outlook not installed / bitness mismatch / corrupted Office registration / orphaned `OUTLOOK.EXE` on the host | **Yes** - 64-bit process vs 32-bit Outlook; host-side fix | -| 2 | Activity times out (`TimeoutMS` elapsed) or job hangs with no exception | Hidden Outlook security prompt, Work Offline, or slow profile load | No - rejected (no timeout/hang; fails fast with a cast) | -| 3 | `NullReferenceException` at the activity | Uninitialized `To`/`Subject`/`Body` or empty attachment path | No - rejected (inputs are valid literals) | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: a `SendOutlookMail` activity with valid literal inputs, `targetFramework "Windows"` (64-bit), dependency `UiPath.Mail.Activities [1.18.3]` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the Branch 1 playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table (`or folders list`, `or jobs list --state Faulted`, `or jobs get`, `or jobs logs --level Error`, `or jobs logs`, plus `docsai ask` passthrough) | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This test **scores the conclusion, not the trajectory.** The -`llm_judge` grades the agent's final response and tool calls against -`RESOLUTION.md`; it does not require any specific investigation path or -internal-state file. - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `send-outlook-mail-failures.md` Branch 1 (COM cast / - library not registered). -- Agent attributed the failure to the Outlook COM registration / - bitness mismatch on the host and handed the user a host-side check - list (matching-bitness Outlook, Office Quick Repair, clear orphaned - `OUTLOOK.EXE`; optionally SMTP/Graph fallback) without fabricating - host actions it could not run. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/aa98a0d3c6.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/aa98a0d3c6.json deleted file mode 100644 index 58622473e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/aa98a0d3c6.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-02T06:15:00.150Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-02T06:15:00.420Z", - "Message": "Robot executor process started (64-bit) for InvoiceMailer under UIPATH\\ROBOTUSER1" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T06:15:01.880Z", - "Message": "[Send Outlook Mail Message] Launching / attaching to Outlook (OUTLOOK.EXE) for the current mail profile" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T06:15:03.401Z", - "Message": "[Send Outlook Mail Message] Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. (Library not registered. Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T06:15:03.452Z", - "Message": "Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T06:15:03.465Z", - "Message": "Execution ended in: 00:00:03" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/b4236c1b60.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/b4236c1b60.json deleted file mode 100644 index 9abedcea53..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/b4236c1b60.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d", - "State": "Faulted", - "ProcessName": "InvoiceMailer", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:15:00.120Z", - "EndTime": "2026-06-02T06:15:03.470Z", - "CreationTime": "2026-06-02T06:14:59.900Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/c52cb02392.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/c52cb02392.json deleted file mode 100644 index 56d392b71d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/c52cb02392.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d", - "StartTime": "2026-06-02T06:15:00.120Z", - "EndTime": "2026-06-02T06:15:03.470Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "c3d4e5f6-7a8b-49c0-8d1e-2f3a4b5c6d7e", - "Info": "Send Outlook Mail Message: Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\r\n\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\r\n ---> System.Runtime.InteropServices.COMException: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))", - "CreationTime": "2026-06-02T06:14:59.900Z", - "ReleaseName": "InvoiceMailer", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 7044218, - "OrganizationUnitId": 8810723, - "FolderKey": "e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-02T06:15:03.470Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d", - "Id": 704421803 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/d5b554c3d7.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/d5b554c3d7.json deleted file mode 100644 index beb82f902a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/d5b554c3d7.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a4b5c6d7-8e9f-4a0b-1c2d-3e4f5a6b7c8d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/manifest.json deleted file mode 100644 index a730e8b47e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "d5b554c3d7.json" - }, - { - "match": "or jobs list --folder-key \"e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c\" --state Faulted --output json", - "file": "b4236c1b60.json" - }, - { - "match": "or jobs list --folder-key e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c --state Faulted --output json", - "file": "b4236c1b60.json" - }, - { - "match": "or jobs get \"f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d\" --output json", - "file": "c52cb02392.json" - }, - { - "match": "or jobs get f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d --output json", - "file": "c52cb02392.json" - }, - { - "match": "or jobs logs \"f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d\" --level Error --output json", - "file": "406f81c8d3.json" - }, - { - "match": "or jobs logs f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d --level Error --output json", - "file": "406f81c8d3.json" - }, - { - "match": "or jobs logs \"f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d\" --output json", - "file": "aa98a0d3c6.json" - }, - { - "match": "or jobs logs f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d --output json", - "file": "aa98a0d3c6.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/uip-fixture.json new file mode 100644 index 0000000000..b7a05ffc69 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:15:00.120Z\",\r\n \"EndTime\": \"2026-06-02T06:15:03.470Z\",\r\n \"CreationTime\": \"2026-06-02T06:14:59.900Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T06:15:03.401Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. (Library not registered. Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T06:15:03.452Z\",\r\n \"Message\": \"Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d\",\r\n \"StartTime\": \"2026-06-02T06:15:00.120Z\",\r\n \"EndTime\": \"2026-06-02T06:15:03.470Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"c3d4e5f6-7a8b-49c0-8d1e-2f3a4b5c6d7e\",\r\n \"Info\": \"Send Outlook Mail Message: Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\\r\\n\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\\r\\n ---> System.Runtime.InteropServices.COMException: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))\",\r\n \"CreationTime\": \"2026-06-02T06:14:59.900Z\",\r\n \"ReleaseName\": \"InvoiceMailer\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 7044218,\r\n \"OrganizationUnitId\": 8810723,\r\n \"FolderKey\": \"e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-02T06:15:03.470Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d\",\r\n \"Id\": 704421803\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f2b3c4d5-6e7f-4a8b-9c0d-1e2f3a4b5c6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T06:15:00.150Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-02T06:15:00.420Z\",\r\n \"Message\": \"Robot executor process started (64-bit) for InvoiceMailer under UIPATH\\\\ROBOTUSER1\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T06:15:01.880Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Launching / attaching to Outlook (OUTLOOK.EXE) for the current mail profile\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T06:15:03.401Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. (Library not registered. Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED))\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T06:15:03.452Z\",\r\n \"Message\": \"Unable to cast COM object of type 'Microsoft.Office.Interop.Outlook.ApplicationClass' to interface type 'Microsoft.Office.Interop.Outlook._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00063001-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T06:15:03.465Z\",\r\n \"Message\": \"Execution ended in: 00:00:03\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e1a2b3c4-5d6e-4f7a-8b9c-0d1e2f3a4b5c\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a4b5c6d7-8e9f-4a0b-1c2d-3e4f5a6b7c8d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/task.yaml index 2c5cda12e6..f8fe5a1041 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-com-not-registered/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/README.md new file mode 100644 index 0000000000..edc96421da --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/README.md @@ -0,0 +1,79 @@ +# Send Outlook Mail Message Failure - NullReferenceException (uninitialized To) + +This scenario reproduces a runtime `Send Outlook Mail Message` failure +caused by an **uninitialized input**: the activity's `To` field is bound +to a variable (`recipient`) that an upstream step never assigned, so it +is `Nothing`/null at runtime. The activity dereferences the null and +faults with +`System.NullReferenceException: Object reference not set to an instance +of an object`. + +## What this scenario uncovers + +**Root Cause:** In `Main.xaml`, `Send Outlook Mail Message` binds +`To="[recipient]"`. The `recipient` variable is declared on the Main +Sequence but is never assigned (the config read that should populate it +is missing), so `To` is null. The activity raises +`NullReferenceException` at `SendOutlookMail`. The empty input is +visible in the job-log Trace just before the fault: +`To='' (empty); Subject='Welcome'`. + +This maps to: +`references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` +(Branch 3 - Uninitialized input). + +The fix is a **workflow bug fixable in Studio** - initialize/populate +`recipient` (and guard the inputs) before the Send, validate by +hardcoding a literal address. It is NOT a host-side fix. + +## Sibling-branch comparison + +The Send Outlook Mail playbook has three cause-branches. Each has its +own scenario so the agent must discriminate on the error signature, not +just match the activity name. + +| Branch | Scenario | Signature | Surface | Fix location | +|--------|----------|-----------|---------|--------------| +| 1 - COM cast / library not registered | `send-outlook-mail-com-not-registered` | `InvalidCastException` / `COMException` (`REGDB_E_CLASSNOTREG` / `TYPE_E_LIBNOTREGISTERED`) | COM layer (Outlook install / bitness) | Host (install/repair Outlook, match bitness) | +| 2 - Timeout / hang | `send-outlook-mail-timeout-security-prompt` | Timeout / hang, no clean inner exception | Session / UI (security prompt, Work Offline) | Host (suppress prompt, go Online, raise TimeoutMS for true slowness) | +| **3 - Uninitialized input (this scenario)** | `send-outlook-mail-null-input` | `NullReferenceException` at the activity | Inputs (null `To`/`Subject`/`Body`) | **Workflow / Studio** (populate the variable, guard inputs) | + +The reject list in the LLM judge rejects any answer that lands on +Branch 1 or Branch 2, on a generic SMTP/network/credentials error, on +an invalid-recipient ("mailbox does not exist") read of the empty +address, or on a host-side fix. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `To="[recipient]"` with `recipient` declared but never assigned in `Main.xaml` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature (Branch 3) rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This test **scores the conclusion, not the trajectory.** The two +required criteria are `skill_triggered` (the `uipath-troubleshoot` skill +activated) and an `llm_judge` graded against `RESOLUTION.md`. The judge +reads only the agent's presented final response and its tool calls - it +does not inspect `.local/investigations/` internal state. An agent that +reaches the right root cause via a different command path still passes. + +The agent passes when it: + +- invoked the `uipath-troubleshoot` skill; +- matched `send-outlook-mail-failures.md` Branch 3 (uninitialized input); +- named the `NullReferenceException` at the Send Outlook Mail Message + activity and attributed it to the uninitialized `recipient` variable + bound to `To` (empty/null input), citing `Main.xaml` and/or the + empty-To job-log Trace; +- recommended a **workflow** fix (initialize/populate `recipient`, guard + the inputs, validate by hardcoding a literal address) - not a + host-side fix. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/0a0d3d14cb.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/0a0d3d14cb.json deleted file mode 100644 index 8223e4d731..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/0a0d3d14cb.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for HR onboarding automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a9b8c7d6-5e4f-4a3b-8c2d-1e0f9a8b7c6d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/128200d21e.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/128200d21e.json deleted file mode 100644 index 1e80c745bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/128200d21e.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012", - "StartTime": "2026-06-02T08:45:00.120Z", - "EndTime": "2026-06-02T08:45:02.190Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "e7f8a9b0-1234-4c5d-8e6f-7a8b9c0d1e2f", - "Info": "Send Outlook Mail Message: Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Mail.Outlook.Activities.SendOutlookMail.Execute(CodeActivityContext context)", - "CreationTime": "2026-06-02T08:44:59.880Z", - "ReleaseName": "OnboardingMailer", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 7042118, - "OrganizationUnitId": 8810427, - "FolderKey": "c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-02T08:45:02.190Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012", - "Id": 749218034 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/README.md deleted file mode 100644 index 81b1ce5cd5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# Send Outlook Mail Message Failure - NullReferenceException (uninitialized To) - -This scenario reproduces a runtime `Send Outlook Mail Message` failure -caused by an **uninitialized input**: the activity's `To` field is bound -to a variable (`recipient`) that an upstream step never assigned, so it -is `Nothing`/null at runtime. The activity dereferences the null and -faults with -`System.NullReferenceException: Object reference not set to an instance -of an object`. - -## What this scenario uncovers - -**Root Cause:** In `Main.xaml`, `Send Outlook Mail Message` binds -`To="[recipient]"`. The `recipient` variable is declared on the Main -Sequence but is never assigned (the config read that should populate it -is missing), so `To` is null. The activity raises -`NullReferenceException` at `SendOutlookMail`. The empty input is -visible in the job-log Trace just before the fault: -`To='' (empty); Subject='Welcome'`. - -This maps to: -`references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` -(Branch 3 - Uninitialized input). - -The fix is a **workflow bug fixable in Studio** - initialize/populate -`recipient` (and guard the inputs) before the Send, validate by -hardcoding a literal address. It is NOT a host-side fix. - -## Sibling-branch comparison - -The Send Outlook Mail playbook has three cause-branches. Each has its -own scenario so the agent must discriminate on the error signature, not -just match the activity name. - -| Branch | Scenario | Signature | Surface | Fix location | -|--------|----------|-----------|---------|--------------| -| 1 - COM cast / library not registered | `send-outlook-mail-com-not-registered` | `InvalidCastException` / `COMException` (`REGDB_E_CLASSNOTREG` / `TYPE_E_LIBNOTREGISTERED`) | COM layer (Outlook install / bitness) | Host (install/repair Outlook, match bitness) | -| 2 - Timeout / hang | `send-outlook-mail-timeout-security-prompt` | Timeout / hang, no clean inner exception | Session / UI (security prompt, Work Offline) | Host (suppress prompt, go Online, raise TimeoutMS for true slowness) | -| **3 - Uninitialized input (this scenario)** | `send-outlook-mail-null-input` | `NullReferenceException` at the activity | Inputs (null `To`/`Subject`/`Body`) | **Workflow / Studio** (populate the variable, guard inputs) | - -The reject list in the LLM judge rejects any answer that lands on -Branch 1 or Branch 2, on a generic SMTP/network/credentials error, on -an invalid-recipient ("mailbox does not exist") read of the empty -address, or on a host-side fix. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `To="[recipient]"` with `recipient` declared but never assigned in `Main.xaml` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table (quoted + unquoted arg variants) | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature (Branch 3) rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This test **scores the conclusion, not the trajectory.** The two -required criteria are `skill_triggered` (the `uipath-troubleshoot` skill -activated) and an `llm_judge` graded against `RESOLUTION.md`. The judge -reads only the agent's presented final response and its tool calls - it -does not inspect `.local/investigations/` internal state. An agent that -reaches the right root cause via a different command path still passes. - -The agent passes when it: - -- invoked the `uipath-troubleshoot` skill; -- matched `send-outlook-mail-failures.md` Branch 3 (uninitialized input); -- named the `NullReferenceException` at the Send Outlook Mail Message - activity and attributed it to the uninitialized `recipient` variable - bound to `To` (empty/null input), citing `Main.xaml` and/or the - empty-To job-log Trace; -- recommended a **workflow** fix (initialize/populate `recipient`, guard - the inputs, validate by hardcoding a literal address) - not a - host-side fix. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/a60eaa5274.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/a60eaa5274.json deleted file mode 100644 index 89779ae18e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/a60eaa5274.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012", - "State": "Faulted", - "ProcessName": "OnboardingMailer", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T08:45:00.120Z", - "EndTime": "2026-06-02T08:45:02.190Z", - "CreationTime": "2026-06-02T08:44:59.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/dc9d2ece7c.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/dc9d2ece7c.json deleted file mode 100644 index c5e4e8ff83..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/dc9d2ece7c.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-02T08:45:00.150Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T08:45:00.820Z", - "Message": "[Log Message] Preparing welcome mail for recipient: " - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-02T08:45:01.640Z", - "Message": "[Send Outlook Mail Message] preparing message: To='' (empty); Subject='Welcome'; Attachments=0" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T08:45:02.120Z", - "Message": "[Send Outlook Mail Message] Object reference not set to an instance of an object." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T08:45:02.165Z", - "Message": "System.NullReferenceException: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T08:45:02.180Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/fff4e04cb0.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/fff4e04cb0.json deleted file mode 100644 index a0b5e71585..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/fff4e04cb0.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-02T08:45:02.120Z", - "Message": "[Send Outlook Mail Message] Object reference not set to an instance of an object." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T08:45:02.165Z", - "Message": "System.NullReferenceException: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/manifest.json deleted file mode 100644 index ba77033e42..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "0a0d3d14cb.json" - }, - { - "match": "or jobs list --folder-key \"c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90\" --state Faulted --output json", - "file": "a60eaa5274.json" - }, - { - "match": "or jobs list --folder-key c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90 --state Faulted --output json", - "file": "a60eaa5274.json" - }, - { - "match": "or jobs get \"d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012\" --output json", - "file": "128200d21e.json" - }, - { - "match": "or jobs get d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012 --output json", - "file": "128200d21e.json" - }, - { - "match": "or jobs logs \"d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012\" --level Error --output json", - "file": "fff4e04cb0.json" - }, - { - "match": "or jobs logs d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012 --level Error --output json", - "file": "fff4e04cb0.json" - }, - { - "match": "or jobs logs \"d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012\" --output json", - "file": "dc9d2ece7c.json" - }, - { - "match": "or jobs logs d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012 --output json", - "file": "dc9d2ece7c.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/uip-fixture.json new file mode 100644 index 0000000000..46af44fca6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OnboardingMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T08:45:00.120Z\",\r\n \"EndTime\": \"2026-06-02T08:45:02.190Z\",\r\n \"CreationTime\": \"2026-06-02T08:44:59.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T08:45:02.120Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Object reference not set to an instance of an object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T08:45:02.165Z\",\r\n \"Message\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012\",\r\n \"StartTime\": \"2026-06-02T08:45:00.120Z\",\r\n \"EndTime\": \"2026-06-02T08:45:02.190Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"e7f8a9b0-1234-4c5d-8e6f-7a8b9c0d1e2f\",\r\n \"Info\": \"Send Outlook Mail Message: Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Mail.Outlook.Activities.SendOutlookMail.Execute(CodeActivityContext context)\",\r\n \"CreationTime\": \"2026-06-02T08:44:59.880Z\",\r\n \"ReleaseName\": \"OnboardingMailer\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 7042118,\r\n \"OrganizationUnitId\": 8810427,\r\n \"FolderKey\": \"c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-02T08:45:02.190Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012\",\r\n \"Id\": 749218034\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d6f7a8b9-0123-4e2f-9a3b-5c6d7e8f9012", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T08:45:00.150Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T08:45:00.820Z\",\r\n \"Message\": \"[Log Message] Preparing welcome mail for recipient: \"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-02T08:45:01.640Z\",\r\n \"Message\": \"[Send Outlook Mail Message] preparing message: To='' (empty); Subject='Welcome'; Attachments=0\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T08:45:02.120Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Object reference not set to an instance of an object.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T08:45:02.165Z\",\r\n \"Message\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T08:45:02.180Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5e6f7a8-9012-4d1e-8f2a-4b5c6d7e8f90\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for HR onboarding automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a9b8c7d6-5e4f-4a3b-8c2d-1e0f9a8b7c6d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/task.yaml index 47bf2dde57..4f0426fdae 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-null-input/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/README.md new file mode 100644 index 0000000000..0325b92f7d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/README.md @@ -0,0 +1,77 @@ +# Send Outlook Mail Message Timeout - Hidden Security Prompt (Branch 2) + +This scenario reproduces a runtime `Send Outlook Mail Message` failure +where the activity **times out / hangs**: on an unattended Robot, the +Outlook COM send triggers the Object Model Guard security prompt ("A +program is trying to send an email message on your behalf") which is +foregrounded but invisible and awaiting input. With no one present on +the unattended session to dismiss it, the call blocks until `TimeoutMS` +(30000 ms) and the job faults with +`System.TimeoutException: The operation has timed out.` + +## What this scenario uncovers + +**Root Cause:** The `SendOutlookMail` activity in `Main.xaml` drives the +Outlook desktop client through COM. On the unattended session (machine +`MOCK-HOST`, account `UIPATH\ROBOTUSER1`) the hidden Object Model Guard +prompt blocks the send until `TimeoutMS` elapses. **Work Offline** on +the Robot's profile is the sibling sub-cause with the same signature. + +This maps to: +`references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` +-- specifically **Branch 2 ("Activity times out or hangs")**. + +The user is framed as **off-host**, so the correct agent behavior is to +hand a host-side check list (ensure a profile exists, turn off Work +Offline, register AV with Windows Security Center or set the +programmatic-access GPO, and/or switch to SMTP/Graph for unattended) +rather than try host commands itself. The real fix is **host-side +and/or design** -- NOT merely raising `TimeoutMS`. + +## Sibling-branch comparison (same playbook, different branch) + +| Branch | Signature | Cause | This scenario? | +|---|---|---|---| +| Branch 1 -- COM cast / library not registered | `InvalidCastException` / `COMException` `REGDB_E_CLASSNOTREG` / `TYPE_E_LIBNOTREGISTERED` | Outlook not installed, bitness mismatch, corrupt type library, orphaned `OUTLOOK.EXE` | No -- Outlook launched + profile loaded; clean timeout, no cast error | +| **Branch 2 -- Activity times out or hangs** | **timeout / hang, `System.TimeoutException`** | **hidden Object Model Guard security prompt or Work Offline on an unattended session; slow first profile load** | **Yes** | +| Branch 3 -- Uninitialized input | `NullReferenceException` at the activity | `To`/`Subject`/`Body` or attachment path is null | No -- inputs are literals in `Main.xaml` | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `SendOutlookMail` with literal `To`/`Subject`/`Body` and `TimeoutMS="30000"` | + +The smoking-gun is in the full job-logs fixture +(`or-jobs-logs-b4d5e6f7-output-json.json`): Info shows Outlook launched +and the profile loaded, then a **Warn** -- "[Send Outlook Mail Message] +Outlook is not responding to the send request; the call has been +pending for >25s (an interactive prompt may be awaiting input on a +non-interactive session)" -- then the timeout Error at exactly +`TimeoutMS`. This steers toward the hidden prompt on the unattended +session without naming it as the answer. + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +This scenario **scores the conclusion, not the trajectory.** The +`llm_judge` grades the agent's final response and tool calls against +`RESOLUTION.md`; it does not require a specific investigation path or +internal `.local/investigations/` state. + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `send-outlook-mail-failures.md` Branch 2 (timeout/hang), + not Branch 1 (COM cast) or Branch 3 (null input). +- Agent attributed the hang to a hidden Outlook security prompt (Object + Model Guard) -- or Work Offline -- on the unattended session. +- Agent handed a host-side check list (profile, Work Offline, AV / + Security-Center registration or programmatic-access GPO, and/or + SMTP/Graph for unattended) and did **not** recommend merely raising + `TimeoutMS`, nor fabricate host commands it could not run. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/054ceeebdd.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/054ceeebdd.json deleted file mode 100644 index e1c8932bf3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/054ceeebdd.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-02T02:30:30.560Z", - "Message": "[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook did not respond to the send request; the COM call blocked until the timeout elapsed." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T02:30:30.601Z", - "Message": "Send Outlook Mail Message: Timeout reached. (30000 ms)\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/1d65f62d80.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/1d65f62d80.json deleted file mode 100644 index 03d404de9d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/1d65f62d80.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f", - "State": "Faulted", - "ProcessName": "NightlyReportMailer", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T02:30:00.140Z", - "EndTime": "2026-06-02T02:30:30.620Z", - "CreationTime": "2026-06-02T02:29:59.910Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/381c939552.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/381c939552.json deleted file mode 100644 index 196c0307ab..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/381c939552.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f", - "StartTime": "2026-06-02T02:30:00.140Z", - "EndTime": "2026-06-02T02:30:30.620Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "ce5f6a71-8293-4a5b-bc6d-7e8f9a0b1c2d", - "Info": "Send Outlook Mail Message: Timeout reached. (30000 ms) Outlook did not respond to the send request; the COM call blocked until the timeout elapsed.\r\n\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out.", - "CreationTime": "2026-06-02T02:29:59.910Z", - "ReleaseName": "NightlyReportMailer", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6029417, - "OrganizationUnitId": 7791618, - "FolderKey": "a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-02T02:30:30.620Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f", - "Id": 691420733 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/7903b9c7a4.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/7903b9c7a4.json deleted file mode 100644 index 8e1a624f49..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/7903b9c7a4.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-02T02:30:00.180Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T02:30:01.402Z", - "Message": "[Send Outlook Mail Message] Attaching to Outlook; launching OUTLOOK.EXE under UIPATH\\ROBOTUSER1" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T02:30:04.713Z", - "Message": "[Send Outlook Mail Message] Outlook profile loaded; composing message to reports-dl@test.com" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-02T02:30:26.190Z", - "Message": "[Send Outlook Mail Message] Outlook is not responding to the send request; the call has been pending for >25s (an interactive prompt may be awaiting input on a non-interactive session)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T02:30:30.560Z", - "Message": "[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook did not respond to the send request; the COM call blocked until the timeout elapsed." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T02:30:30.601Z", - "Message": "Send Outlook Mail Message: Timeout reached. (30000 ms)\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T02:30:30.615Z", - "Message": "Execution ended in: 00:00:30" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/836b988d1c.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/836b988d1c.json deleted file mode 100644 index 203ff533ba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/836b988d1c.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/README.md deleted file mode 100644 index 475b97c866..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/README.md +++ /dev/null @@ -1,76 +0,0 @@ -# Send Outlook Mail Message Timeout - Hidden Security Prompt (Branch 2) - -This scenario reproduces a runtime `Send Outlook Mail Message` failure -where the activity **times out / hangs**: on an unattended Robot, the -Outlook COM send triggers the Object Model Guard security prompt ("A -program is trying to send an email message on your behalf") which is -foregrounded but invisible and awaiting input. With no one present on -the unattended session to dismiss it, the call blocks until `TimeoutMS` -(30000 ms) and the job faults with -`System.TimeoutException: The operation has timed out.` - -## What this scenario uncovers - -**Root Cause:** The `SendOutlookMail` activity in `Main.xaml` drives the -Outlook desktop client through COM. On the unattended session (machine -`MOCK-HOST`, account `UIPATH\ROBOTUSER1`) the hidden Object Model Guard -prompt blocks the send until `TimeoutMS` elapses. **Work Offline** on -the Robot's profile is the sibling sub-cause with the same signature. - -This maps to: -`references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` --- specifically **Branch 2 ("Activity times out or hangs")**. - -The user is framed as **off-host**, so the correct agent behavior is to -hand a host-side check list (ensure a profile exists, turn off Work -Offline, register AV with Windows Security Center or set the -programmatic-access GPO, and/or switch to SMTP/Graph for unattended) -rather than try host commands itself. The real fix is **host-side -and/or design** -- NOT merely raising `TimeoutMS`. - -## Sibling-branch comparison (same playbook, different branch) - -| Branch | Signature | Cause | This scenario? | -|---|---|---|---| -| Branch 1 -- COM cast / library not registered | `InvalidCastException` / `COMException` `REGDB_E_CLASSNOTREG` / `TYPE_E_LIBNOTREGISTERED` | Outlook not installed, bitness mismatch, corrupt type library, orphaned `OUTLOOK.EXE` | No -- Outlook launched + profile loaded; clean timeout, no cast error | -| **Branch 2 -- Activity times out or hangs** | **timeout / hang, `System.TimeoutException`** | **hidden Object Model Guard security prompt or Work Offline on an unattended session; slow first profile load** | **Yes** | -| Branch 3 -- Uninitialized input | `NullReferenceException` at the activity | `To`/`Subject`/`Body` or attachment path is null | No -- inputs are literals in `Main.xaml` | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `SendOutlookMail` with literal `To`/`Subject`/`Body` and `TimeoutMS="30000"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook Branch 2 signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The smoking-gun is in the full job-logs fixture -(`or-jobs-logs-b4d5e6f7-output-json.json`): Info shows Outlook launched -and the profile loaded, then a **Warn** -- "[Send Outlook Mail Message] -Outlook is not responding to the send request; the call has been -pending for >25s (an interactive prompt may be awaiting input on a -non-interactive session)" -- then the timeout Error at exactly -`TimeoutMS`. This steers toward the hidden prompt on the unattended -session without naming it as the answer. - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -This scenario **scores the conclusion, not the trajectory.** The -`llm_judge` grades the agent's final response and tool calls against -`RESOLUTION.md`; it does not require a specific investigation path or -internal `.local/investigations/` state. - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `send-outlook-mail-failures.md` Branch 2 (timeout/hang), - not Branch 1 (COM cast) or Branch 3 (null input). -- Agent attributed the hang to a hidden Outlook security prompt (Object - Model Guard) -- or Work Offline -- on the unattended session. -- Agent handed a host-side check list (profile, Work Offline, AV / - Security-Center registration or programmatic-access GPO, and/or - SMTP/Graph for unattended) and did **not** recommend merely raising - `TimeoutMS`, nor fabricate host commands it could not run. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/manifest.json deleted file mode 100644 index 534bd6ad5c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send Outlook Mail Message scenario; mock dispatcher only -- do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "836b988d1c.json" - }, - { - "match": "or jobs list --folder-key \"a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e\" --state Faulted --output json", - "file": "1d65f62d80.json" - }, - { - "match": "or jobs list --folder-key a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e --state Faulted --output json", - "file": "1d65f62d80.json" - }, - { - "match": "or jobs get \"b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f\" --output json", - "file": "381c939552.json" - }, - { - "match": "or jobs get b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f --output json", - "file": "381c939552.json" - }, - { - "match": "or jobs logs \"b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f\" --level Error --output json", - "file": "054ceeebdd.json" - }, - { - "match": "or jobs logs b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f --level Error --output json", - "file": "054ceeebdd.json" - }, - { - "match": "or jobs logs \"b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f\" --output json", - "file": "7903b9c7a4.json" - }, - { - "match": "or jobs logs b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f --output json", - "file": "7903b9c7a4.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/uip-fixture.json new file mode 100644 index 0000000000..649cf8bc13 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReportMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T02:30:00.140Z\",\r\n \"EndTime\": \"2026-06-02T02:30:30.620Z\",\r\n \"CreationTime\": \"2026-06-02T02:29:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T02:30:30.560Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook did not respond to the send request; the COM call blocked until the timeout elapsed.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T02:30:30.601Z\",\r\n \"Message\": \"Send Outlook Mail Message: Timeout reached. (30000 ms)\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f\",\r\n \"StartTime\": \"2026-06-02T02:30:00.140Z\",\r\n \"EndTime\": \"2026-06-02T02:30:30.620Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"ce5f6a71-8293-4a5b-bc6d-7e8f9a0b1c2d\",\r\n \"Info\": \"Send Outlook Mail Message: Timeout reached. (30000 ms) Outlook did not respond to the send request; the COM call blocked until the timeout elapsed.\\r\\n\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\",\r\n \"CreationTime\": \"2026-06-02T02:29:59.910Z\",\r\n \"ReleaseName\": \"NightlyReportMailer\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6029417,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-02T02:30:30.620Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f\",\r\n \"Id\": 691420733\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b4d5e6f7-8091-4c0d-9e1f-3a4b5c6d7e8f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T02:30:00.180Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T02:30:01.402Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Attaching to Outlook; launching OUTLOOK.EXE under UIPATH\\\\ROBOTUSER1\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T02:30:04.713Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Outlook profile loaded; composing message to reports-dl@test.com\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-02T02:30:26.190Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Outlook is not responding to the send request; the call has been pending for >25s (an interactive prompt may be awaiting input on a non-interactive session)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T02:30:30.560Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook did not respond to the send request; the COM call blocked until the timeout elapsed.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T02:30:30.601Z\",\r\n \"Message\": \"Send Outlook Mail Message: Timeout reached. (30000 ms)\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T02:30:30.615Z\",\r\n \"Message\": \"Execution ended in: 00:00:30\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a3c4d5e6-7f80-4b9c-8d0e-2f3a4b5c6d7e\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/task.yaml index f278c20f59..02c2ac357c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-timeout-security-prompt/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/README.md new file mode 100644 index 0000000000..bdcd9492d5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/README.md @@ -0,0 +1,68 @@ +# Send Outlook Mail Message Timeout — Work Offline (Branch 2) + +Runtime troubleshooting scenario for `UiPath.Mail.Outlook.Activities` `Send Outlook Mail Message` +(`SendOutlookMail`), covering the **Work Offline** sub-cause of +`send-outlook-mail-failures.md` **Branch 2 ("Activity times out or hangs")**. + +## What this scenario exercises + +An unattended job faults with `System.TimeoutException` after `TimeoutMS` (30000 ms). Outlook launched +and the profile loaded, the message was composed and **placed in the Outbox**, but it was never +transmitted because Outlook is **offline (Work Offline)** — so the COM send blocks until the timeout. +The agent must match Branch 2 and attribute the hang specifically to **Work Offline / an offline +mailbox**, and hand a host-side check list (turn off `Send/Receive > Work Offline`, confirm the profile +connects Online, flush the Outbox; switch to SMTP/Graph for unattended). It must NOT diagnose the +security-prompt sibling, a COM-cast (Branch 1), or a null input (Branch 3), and must NOT merely raise +`TimeoutMS`. + +## The discriminator vs the security-prompt sibling + +Branch 2 has two sub-causes with the **same Orchestrator signature** (timeout at `TimeoutMS`, no COM +error, no null): a hidden Object Model Guard **security prompt** and **Work Offline**. This scenario is +distinguished from `mail-send-outlook-mail-timeout-security-prompt` by evidence that rules the prompt +out and rules Work Offline in: + +- Job log: `item placed in the Outbox for sending` → `still in the Outbox and has not been transmitted; + Outlook connectivity state reports Disconnected ... (no dialog or prompt was raised)`. +- User's attended re-run on the host: **no popup/dialog appeared**, and the email was left **unsent in + the Outbox**. + +A "hidden security prompt" answer should score lower here because the evidence explicitly shows no +prompt and a queued Outbox item — the offline fingerprint. + +## Sibling-branch comparison (same playbook) + +| Branch / sub-cause | Signature | This scenario? | +|---|---|---| +| Branch 1 — COM cast / library not registered | `InvalidCastException` / `COMException` `REGDB_E_CLASSNOTREG` | No — Outlook launched + profile loaded; clean timeout | +| Branch 2 — **security prompt** (Object Model Guard) | timeout / hang; a modal awaiting input | No — sibling scenario; here no prompt was raised | +| **Branch 2 — Work Offline** | **timeout / hang; item queued in the Outbox, Outlook offline, no prompt** | **Yes** | +| Branch 3 — Uninitialized input | `NullReferenceException` at the activity | No — `To`/`Subject`/`Body` are literals | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `SendOutlookMail` with literal `To`/`Subject`/`Body` and `TimeoutMS="30000"` on an unattended process | + +The user is framed as **off-host** (Orchestrator only), so the correct behavior is to hand a host-side +check list and stop, not to run host commands. Diagnosis is not disclosed in any agent-visible name +(project `NightlyStatementMailer`, activity "Send Outlook Mail Message"); the prompt states only +observed symptoms (timeout on every run; no popup in an attended test; email left in the Outbox). + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched Branch 2 and attributed the hang to **Work Offline / an offline mailbox** (item queued + in the Outbox, no prompt) — not the security prompt, COM cast, or null input. +- Agent handed a host-side check list (turn off Work Offline, confirm the profile connects Online, flush + the Outbox; SMTP/Graph for unattended) and did NOT recommend merely raising `TimeoutMS`. + +Playbook: `references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` +(Branch 2 — Work Offline sub-cause). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/3db6b96527.json deleted file mode 100644 index 25c26ac8c1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/3db6b96527.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d1e2f3a4-5b6c-4d7e-9f80-1a2b3c4d5e6f", - "StartTime": "2026-06-03T02:30:00.160Z", - "EndTime": "2026-06-03T02:30:30.740Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "e2f3a4b5-6c7d-4e8f-a091-2b3c4d5e6f70", - "Info": "Send Outlook Mail Message: Timeout reached. (30000 ms) Outlook accepted the item into the Outbox but did not transmit it before the timeout elapsed; the COM send did not complete.\r\n\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out.", - "CreationTime": "2026-06-03T02:29:59.920Z", - "ReleaseName": "NightlyStatementMailer", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6039418, - "OrganizationUnitId": 7791618, - "FolderKey": "c9d0e1f2-3a4b-4c5d-8e6f-7a8b9c0d1e2f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-03T02:30:30.740Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d1e2f3a4-5b6c-4d7e-9f80-1a2b3c4d5e6f", - "Id": 691530844 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/75949d659e.json deleted file mode 100644 index ba7ee631fe..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/75949d659e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d1e2f3a4-5b6c-4d7e-9f80-1a2b3c4d5e6f", - "State": "Faulted", - "ProcessName": "NightlyStatementMailer", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T02:30:00.160Z", - "EndTime": "2026-06-03T02:30:30.740Z", - "CreationTime": "2026-06-03T02:29:59.920Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/README.md deleted file mode 100644 index 8d9ba3ff0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/README.md +++ /dev/null @@ -1,66 +0,0 @@ -# Send Outlook Mail Message Timeout — Work Offline (Branch 2) - -Runtime troubleshooting scenario for `UiPath.Mail.Outlook.Activities` `Send Outlook Mail Message` -(`SendOutlookMail`), covering the **Work Offline** sub-cause of -`send-outlook-mail-failures.md` **Branch 2 ("Activity times out or hangs")**. - -## What this scenario exercises - -An unattended job faults with `System.TimeoutException` after `TimeoutMS` (30000 ms). Outlook launched -and the profile loaded, the message was composed and **placed in the Outbox**, but it was never -transmitted because Outlook is **offline (Work Offline)** — so the COM send blocks until the timeout. -The agent must match Branch 2 and attribute the hang specifically to **Work Offline / an offline -mailbox**, and hand a host-side check list (turn off `Send/Receive > Work Offline`, confirm the profile -connects Online, flush the Outbox; switch to SMTP/Graph for unattended). It must NOT diagnose the -security-prompt sibling, a COM-cast (Branch 1), or a null input (Branch 3), and must NOT merely raise -`TimeoutMS`. - -## The discriminator vs the security-prompt sibling - -Branch 2 has two sub-causes with the **same Orchestrator signature** (timeout at `TimeoutMS`, no COM -error, no null): a hidden Object Model Guard **security prompt** and **Work Offline**. This scenario is -distinguished from `mail-send-outlook-mail-timeout-security-prompt` by evidence that rules the prompt -out and rules Work Offline in: - -- Job log: `item placed in the Outbox for sending` → `still in the Outbox and has not been transmitted; - Outlook connectivity state reports Disconnected ... (no dialog or prompt was raised)`. -- User's attended re-run on the host: **no popup/dialog appeared**, and the email was left **unsent in - the Outbox**. - -A "hidden security prompt" answer should score lower here because the evidence explicitly shows no -prompt and a queued Outbox item — the offline fingerprint. - -## Sibling-branch comparison (same playbook) - -| Branch / sub-cause | Signature | This scenario? | -|---|---|---| -| Branch 1 — COM cast / library not registered | `InvalidCastException` / `COMException` `REGDB_E_CLASSNOTREG` | No — Outlook launched + profile loaded; clean timeout | -| Branch 2 — **security prompt** (Object Model Guard) | timeout / hang; a modal awaiting input | No — sibling scenario; here no prompt was raised | -| **Branch 2 — Work Offline** | **timeout / hang; item queued in the Outbox, Outlook offline, no prompt** | **Yes** | -| Branch 3 — Uninitialized input | `NullReferenceException` at the activity | No — `To`/`Subject`/`Body` are literals | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `SendOutlookMail` with literal `To`/`Subject`/`Body` and `TimeoutMS="30000"` on an unattended process | -| `data/m/r/` | synthetic canned `uip` responses; the smoking gun is in `job-logs.json` (item placed in Outbox → still in Outbox / Disconnected / no prompt → timeout at `TimeoutMS`); `docsai ask` passthrough | - -The user is framed as **off-host** (Orchestrator only), so the correct behavior is to hand a host-side -check list and stop, not to run host commands. Diagnosis is not leaked in any agent-visible name -(project `NightlyStatementMailer`, activity "Send Outlook Mail Message"); the prompt states only -observed symptoms (timeout on every run; no popup in an attended test; email left in the Outbox). - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched Branch 2 and attributed the hang to **Work Offline / an offline mailbox** (item queued - in the Outbox, no prompt) — not the security prompt, COM cast, or null input. -- Agent handed a host-side check list (turn off Work Offline, confirm the profile connects Online, flush - the Outbox; SMTP/Graph for unattended) and did NOT recommend merely raising `TimeoutMS`. - -Playbook: `references/activity-packages/mail-activities/playbooks/send-outlook-mail-failures.md` -(Branch 2 — Work Offline sub-cause). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/b8d18266dc.json deleted file mode 100644 index 70f8db3812..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:30:00.200Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:30:01.420Z", - "Message": "[Send Outlook Mail Message] Attaching to Outlook; launching OUTLOOK.EXE under UIPATH\\ROBOTUSER1" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:30:04.760Z", - "Message": "[Send Outlook Mail Message] Outlook profile loaded; composing message to statements-dl@test.com" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:30:05.910Z", - "Message": "[Send Outlook Mail Message] Message composed; item placed in the Outbox for sending" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-03T02:30:26.240Z", - "Message": "[Send Outlook Mail Message] The item is still in the Outbox and has not been transmitted; Outlook connectivity state reports Disconnected. Send pending >20s (no dialog or prompt was raised)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:30:30.680Z", - "Message": "[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook accepted the item into the Outbox but did not transmit it before the timeout elapsed; the COM send did not complete." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:30:30.712Z", - "Message": "Send Outlook Mail Message: Timeout reached. (30000 ms)\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-03T02:30:30.730Z", - "Message": "Execution ended in: 00:00:30" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/d742317695.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/d742317695.json deleted file mode 100644 index ac478deccd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/d742317695.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:30:30.680Z", - "Message": "[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook accepted the item into the Outbox but did not transmit it before the timeout elapsed; the COM send did not complete." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-03T02:30:30.712Z", - "Message": "Send Outlook Mail Message: Timeout reached. (30000 ms)\r\nin Main.xaml\r\n at SendOutlookMail \"Send Outlook Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.TimeoutException: The operation has timed out." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/ee2628e6b6.json deleted file mode 100644 index 315dee9716..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c9d0e1f2-3a4b-4c5d-8e6f-7a8b9c0d1e2f", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/manifest.json deleted file mode 100644 index c2100c078e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/m/r/manifest.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send Outlook Mail Message Work-Offline scenario; mock dispatcher only -- do not treat as investigation evidence. Same timeout signature as the security-prompt sibling, but the evidence (message queued in the Outbox, no security prompt) points to Work Offline. docsai proxied to the real CLI; the --level Error logs rule precedes the generic logs rule. Matching is token-aware (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs logs --level Error", - "file": "d742317695.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/uip-fixture.json new file mode 100644 index 0000000000..f0471f3ac5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:30:30.680Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook accepted the item into the Outbox but did not transmit it before the timeout elapsed; the COM send did not complete.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:30:30.712Z\",\r\n \"Message\": \"Send Outlook Mail Message: Timeout reached. (30000 ms)\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c9d0e1f2-3a4b-4c5d-8e6f-7a8b9c0d1e2f\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d1e2f3a4-5b6c-4d7e-9f80-1a2b3c4d5e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyStatementMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T02:30:00.160Z\",\r\n \"EndTime\": \"2026-06-03T02:30:30.740Z\",\r\n \"CreationTime\": \"2026-06-03T02:29:59.920Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:30:00.200Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:30:01.420Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Attaching to Outlook; launching OUTLOOK.EXE under UIPATH\\\\ROBOTUSER1\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:30:04.760Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Outlook profile loaded; composing message to statements-dl@test.com\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:30:05.910Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Message composed; item placed in the Outbox for sending\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-03T02:30:26.240Z\",\r\n \"Message\": \"[Send Outlook Mail Message] The item is still in the Outbox and has not been transmitted; Outlook connectivity state reports Disconnected. Send pending >20s (no dialog or prompt was raised)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:30:30.680Z\",\r\n \"Message\": \"[Send Outlook Mail Message] Timeout reached. (30000 ms) Outlook accepted the item into the Outbox but did not transmit it before the timeout elapsed; the COM send did not complete.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-03T02:30:30.712Z\",\r\n \"Message\": \"Send Outlook Mail Message: Timeout reached. (30000 ms)\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-03T02:30:30.730Z\",\r\n \"Message\": \"Execution ended in: 00:00:30\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d1e2f3a4-5b6c-4d7e-9f80-1a2b3c4d5e6f\",\r\n \"StartTime\": \"2026-06-03T02:30:00.160Z\",\r\n \"EndTime\": \"2026-06-03T02:30:30.740Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"e2f3a4b5-6c7d-4e8f-a091-2b3c4d5e6f70\",\r\n \"Info\": \"Send Outlook Mail Message: Timeout reached. (30000 ms) Outlook accepted the item into the Outbox but did not transmit it before the timeout elapsed; the COM send did not complete.\\r\\n\\r\\nin Main.xaml\\r\\n at SendOutlookMail \\\"Send Outlook Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.TimeoutException: The operation has timed out.\",\r\n \"CreationTime\": \"2026-06-03T02:29:59.920Z\",\r\n \"ReleaseName\": \"NightlyStatementMailer\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6039418,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"c9d0e1f2-3a4b-4c5d-8e6f-7a8b9c0d1e2f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-03T02:30:30.740Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d1e2f3a4-5b6c-4d7e-9f80-1a2b3c4d5e6f\",\r\n \"Id\": 691530844\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/task.yaml index 515a77cf3d..1accc1e770 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-outlook-mail-work-offline/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/README.md new file mode 100644 index 0000000000..ac214c1f63 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/README.md @@ -0,0 +1,48 @@ +# Send SMTP Mail Message — Basic Auth / SMTP AUTH Disabled (Branch 2) + +Runtime troubleshooting scenario for `UiPath.Mail.SMTP.Activities` `Send SMTP Mail Message` +(`SmtpSendMail`), covering the **Basic-Auth / SMTP-AUTH-disabled** sub-case of +`send-smtp-mail-failures.md` **Branch 2**. + +## What this scenario exercises + +An unattended job faults with `System.Net.Mail.SmtpException ... 535 5.7.139 Authentication +unsuccessful ... [basic authentication is disabled]`. The activity connected to `smtp.office365.com:587` +and negotiated STARTTLS fine — Microsoft 365 refused the **Basic-Auth** login because SMTP AUTH is +disabled for the account. The agent must recognize the `5.7.139` code as a **disabled auth mechanism** +(not a wrong password, not a transport problem) and prescribe moving to OAuth2 / Graph o365 (preferred) +or enabling SMTP AUTH on the mailbox — NOT rotating the password. + +## Discriminator vs the other auth/connection branches + +- `5.7.139` (mechanism disabled) — **this scenario**; the fix is auth-policy / OAuth, not the secret. +- `535 5.7.3` (secret rejected) — Branch 1, a real credential problem. +- `SocketException` / TLS handshake — Branch 3, a Port↔`SecureConnection` mispairing. Here STARTTLS on + 587 succeeded, so it is not Branch 3. + +The logs make the discriminator explicit: STARTTLS negotiated and `AUTH LOGIN` was attempted, then the +server returned `5.7.139 ... basic authentication is disabled` — transport OK, mechanism refused. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `SmtpSendMail` targeting `smtp.office365.com:587` StartTls with Basic-Auth `Email` | + +Diagnosis is not disclosed in any agent-visible name (project `AutomatedBillingNotifier`, activity "Send +SMTP Mail Message"); the prompt states only the observed symptom (job faults with the pasted `535` +error). + +## Success criteria + +`skill_triggered` + `llm_judge` against `RESOLUTION.md`: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the disabled Basic-Auth / SMTP-AUTH mechanism (`5.7.139`) as the cause and prescribed + OAuth2 / Graph o365 or enabling SMTP AUTH — not a password rotation, transport change, or Outlook swap. + +Playbook: `references/activity-packages/mail-activities/playbooks/send-smtp-mail-failures.md` (Branch 2). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/3db6b96527.json deleted file mode 100644 index 7d8d5299fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/3db6b96527.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a4b5c6d7-8293-4c04-9d5e-6f7081920415", - "StartTime": "2026-06-05T05:00:00.180Z", - "EndTime": "2026-06-05T05:00:03.010Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "b5c6d7e8-9304-4d15-8e6f-708192a30516", - "Info": "Send SMTP Mail Message: Failure sending mail.\r\n\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 535 5.7.139 Authentication unsuccessful, the request did not meet the criteria to be authenticated successfully. Contact your administrator. [basic authentication is disabled]", - "CreationTime": "2026-06-05T04:59:59.910Z", - "ReleaseName": "AutomatedBillingNotifier", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6051407, - "OrganizationUnitId": 7791618, - "FolderKey": "d3e4f5a6-7182-4b93-9c4d-5e6f70819304", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-05T05:00:03.010Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a4b5c6d7-8293-4c04-9d5e-6f7081920415", - "Id": 682611995 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/75949d659e.json deleted file mode 100644 index 6e4b8dcee3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/75949d659e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a4b5c6d7-8293-4c04-9d5e-6f7081920415", - "State": "Faulted", - "ProcessName": "AutomatedBillingNotifier", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-05T05:00:00.180Z", - "EndTime": "2026-06-05T05:00:03.010Z", - "CreationTime": "2026-06-05T04:59:59.910Z" - } - ], - "Pagination": { "Returned": 1, "Limit": 50, "Offset": 0, "HasMore": false } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/README.md deleted file mode 100644 index f4251db04e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Send SMTP Mail Message — Basic Auth / SMTP AUTH Disabled (Branch 2) - -Runtime troubleshooting scenario for `UiPath.Mail.SMTP.Activities` `Send SMTP Mail Message` -(`SmtpSendMail`), covering the **Basic-Auth / SMTP-AUTH-disabled** sub-case of -`send-smtp-mail-failures.md` **Branch 2**. - -## What this scenario exercises - -An unattended job faults with `System.Net.Mail.SmtpException ... 535 5.7.139 Authentication -unsuccessful ... [basic authentication is disabled]`. The activity connected to `smtp.office365.com:587` -and negotiated STARTTLS fine — Microsoft 365 refused the **Basic-Auth** login because SMTP AUTH is -disabled for the account. The agent must recognize the `5.7.139` code as a **disabled auth mechanism** -(not a wrong password, not a transport problem) and prescribe moving to OAuth2 / Graph o365 (preferred) -or enabling SMTP AUTH on the mailbox — NOT rotating the password. - -## Discriminator vs the other auth/connection branches - -- `5.7.139` (mechanism disabled) — **this scenario**; the fix is auth-policy / OAuth, not the secret. -- `535 5.7.3` (secret rejected) — Branch 1, a real credential problem. -- `SocketException` / TLS handshake — Branch 3, a Port↔`SecureConnection` mispairing. Here STARTTLS on - 587 succeeded, so it is not Branch 3. - -The logs make the discriminator explicit: STARTTLS negotiated and `AUTH LOGIN` was attempted, then the -server returned `5.7.139 ... basic authentication is disabled` — transport OK, mechanism refused. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `SmtpSendMail` targeting `smtp.office365.com:587` StartTls with Basic-Auth `Email` | -| `data/m/r/` | synthetic canned `uip` responses; the smoking gun is `job-logs.json` (connect → STARTTLS → AUTH LOGIN → `535 5.7.139 basic authentication is disabled`); `docsai ask` passthrough | - -Diagnosis is not leaked in any agent-visible name (project `AutomatedBillingNotifier`, activity "Send -SMTP Mail Message"); the prompt states only the observed symptom (job faults with the pasted `535` -error). - -## Success criteria - -`skill_triggered` + `llm_judge` against `RESOLUTION.md`: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the disabled Basic-Auth / SMTP-AUTH mechanism (`5.7.139`) as the cause and prescribed - OAuth2 / Graph o365 or enabling SMTP AUTH — not a password rotation, transport change, or Outlook swap. - -Playbook: `references/activity-packages/mail-activities/playbooks/send-smtp-mail-failures.md` (Branch 2). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/b8d18266dc.json deleted file mode 100644 index 93ae8c9d9e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { "Level": "Info", "TimeStamp": "2026-06-05T05:00:00.220Z", "Message": "Execution started" }, - { "Level": "Trace", "TimeStamp": "2026-06-05T05:00:01.020Z", "Message": "[Send SMTP Mail Message] Connecting to smtp.office365.com:587 (SecureConnection=StartTls)" }, - { "Level": "Trace", "TimeStamp": "2026-06-05T05:00:01.640Z", "Message": "[Send SMTP Mail Message] STARTTLS negotiated; authenticating as svc-billing@contoso-test.com (Basic Auth / AUTH LOGIN)" }, - { "Level": "Error", "TimeStamp": "2026-06-05T05:00:02.760Z", "Message": "[Send SMTP Mail Message] Server rejected authentication: 535 5.7.139 Authentication unsuccessful, the request did not meet the criteria to be authenticated successfully. Contact your administrator. [basic authentication is disabled]" }, - { "Level": "Error", "TimeStamp": "2026-06-05T05:00:02.900Z", "Message": "Send SMTP Mail Message: Failure sending mail.\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 535 5.7.139 Authentication unsuccessful ... [basic authentication is disabled]" }, - { "Level": "Info", "TimeStamp": "2026-06-05T05:00:02.990Z", "Message": "Execution ended in: 00:00:03" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/d742317695.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/d742317695.json deleted file mode 100644 index e091f60d0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/d742317695.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { "Level": "Error", "TimeStamp": "2026-06-05T05:00:02.760Z", "Message": "[Send SMTP Mail Message] Server rejected authentication: 535 5.7.139 Authentication unsuccessful, the request did not meet the criteria to be authenticated successfully. Contact your administrator. [basic authentication is disabled]" }, - { "Level": "Error", "TimeStamp": "2026-06-05T05:00:02.900Z", "Message": "Send SMTP Mail Message: Failure sending mail.\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 535 5.7.139 Authentication unsuccessful ... [basic authentication is disabled]" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/ee2628e6b6.json deleted file mode 100644 index 9606db2f91..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d3e4f5a6-7182-4b93-9c4d-5e6f70819304", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/manifest.json deleted file mode 100644 index 67acd0d505..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/m/r/manifest.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send SMTP Mail Message auth / Basic-Auth-disabled sub-case (Branch 2 of send-smtp-mail-failures); mock dispatcher only. The M365 server rejects the Basic-Auth SMTP login with 535 5.7.139 basic authentication is disabled — the credentials are correct; the auth mechanism is off. docsai proxied; --level Error logs rule precedes the generic logs rule. Token-aware matching, first match wins.", - "rules": [ - { "match": "docsai ask", "passthrough": true }, - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs logs --level Error", "file": "d742317695.json" }, - { "match": "or jobs logs", "file": "b8d18266dc.json" }, - { "match": "or jobs get", "file": "3db6b96527.json" } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0, "_doc": "Permissive fallback for unmatched commands." } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/uip-fixture.json new file mode 100644 index 0000000000..f2ec06d10d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-05T05:00:02.760Z\", \"Message\": \"[Send SMTP Mail Message] Server rejected authentication: 535 5.7.139 Authentication unsuccessful, the request did not meet the criteria to be authenticated successfully. Contact your administrator. [basic authentication is disabled]\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-05T05:00:02.900Z\", \"Message\": \"Send SMTP Mail Message: Failure sending mail.\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 535 5.7.139 Authentication unsuccessful ... [basic authentication is disabled]\" }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d3e4f5a6-7182-4b93-9c4d-5e6f70819304\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a4b5c6d7-8293-4c04-9d5e-6f7081920415\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AutomatedBillingNotifier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-05T05:00:00.180Z\",\r\n \"EndTime\": \"2026-06-05T05:00:03.010Z\",\r\n \"CreationTime\": \"2026-06-05T04:59:59.910Z\"\r\n }\r\n ],\r\n \"Pagination\": { \"Returned\": 1, \"Limit\": 50, \"Offset\": 0, \"HasMore\": false }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n { \"Level\": \"Info\", \"TimeStamp\": \"2026-06-05T05:00:00.220Z\", \"Message\": \"Execution started\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-05T05:00:01.020Z\", \"Message\": \"[Send SMTP Mail Message] Connecting to smtp.office365.com:587 (SecureConnection=StartTls)\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-05T05:00:01.640Z\", \"Message\": \"[Send SMTP Mail Message] STARTTLS negotiated; authenticating as svc-billing@contoso-test.com (Basic Auth / AUTH LOGIN)\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-05T05:00:02.760Z\", \"Message\": \"[Send SMTP Mail Message] Server rejected authentication: 535 5.7.139 Authentication unsuccessful, the request did not meet the criteria to be authenticated successfully. Contact your administrator. [basic authentication is disabled]\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-05T05:00:02.900Z\", \"Message\": \"Send SMTP Mail Message: Failure sending mail.\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 535 5.7.139 Authentication unsuccessful ... [basic authentication is disabled]\" },\r\n { \"Level\": \"Info\", \"TimeStamp\": \"2026-06-05T05:00:02.990Z\", \"Message\": \"Execution ended in: 00:00:03\" }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a4b5c6d7-8293-4c04-9d5e-6f7081920415\",\r\n \"StartTime\": \"2026-06-05T05:00:00.180Z\",\r\n \"EndTime\": \"2026-06-05T05:00:03.010Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"b5c6d7e8-9304-4d15-8e6f-708192a30516\",\r\n \"Info\": \"Send SMTP Mail Message: Failure sending mail.\\r\\n\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 535 5.7.139 Authentication unsuccessful, the request did not meet the criteria to be authenticated successfully. Contact your administrator. [basic authentication is disabled]\",\r\n \"CreationTime\": \"2026-06-05T04:59:59.910Z\",\r\n \"ReleaseName\": \"AutomatedBillingNotifier\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6051407,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"d3e4f5a6-7182-4b93-9c4d-5e6f70819304\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-05T05:00:03.010Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a4b5c6d7-8293-4c04-9d5e-6f7081920415\",\r\n \"Id\": 682611995\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/task.yaml index 700ada5b57..1a1be88893 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-auth-basic-disabled/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/README.md new file mode 100644 index 0000000000..debe6bc8d2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/README.md @@ -0,0 +1,47 @@ +# Send SMTP Mail Message — Port / Transport-Security Mismatch (Branch 3) + +Runtime troubleshooting scenario for `UiPath.Mail.SMTP.Activities` `Send SMTP Mail Message` +(`SmtpSendMail`), covering the **connection / port-vs-`SecureConnection` mismatch** of +`send-smtp-mail-failures.md` **Branch 3**. + +## What this scenario exercises + +An unattended job hangs ~35 s then faults with `System.Net.Mail.SmtpException: Failure sending mail` +wrapping an `IOException` → `SocketException` ("An existing connection was forcibly closed"). The +activity pairs `Port=465` with `SecureConnection=StartTls`, but 465 expects **implicit** SSL — so the +STARTTLS negotiation fails and the server drops the socket. The agent must attribute the failure to the +Port↔`SecureConnection` mismatch (not auth, not relay, not a server outage) and prescribe the correct +pairing: **587 + StartTls** or **465 + SSL**. + +## Discriminator + +- The socket **connected** (`TCP connected`) and the failure is at the **TLS negotiation** — so it is + not a server-unreachable outage. +- There is **no `535`/`5.7.x` auth code** and **no `550`/`5.7.1` relay code** — so it is not Branch 1/2 + (auth) or Branch 4/5 (relay/recipient). The error is a `SocketException` at transport setup. +- Source shows the decisive mispairing: `Port="465"` + `SecureConnection="StartTls"`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `SmtpSendMail` with `Server="smtp.office365.com"`, `Port="465"`, `SecureConnection="StartTls"` | + +Diagnosis is not disclosed in any agent-visible name (project `WeeklyReportDispatcher`, activity "Send SMTP +Mail Message"); the prompt states only the observed symptom (job hangs then faults with the pasted +`Failure sending mail` / socket error). + +## Success criteria + +`skill_triggered` + `llm_judge` against `RESOLUTION.md`: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the Port↔`SecureConnection` mismatch (465 + StartTls) as the cause and prescribed the + correct pairing (587 + StartTls, or 465 + SSL) — not an auth, relay, or server-outage misdiagnosis, + and not disabling `SecureConnection`. + +Playbook: `references/activity-packages/mail-activities/playbooks/send-smtp-mail-failures.md` (Branch 3). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/3db6b96527.json deleted file mode 100644 index 441eff8c3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/3db6b96527.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b6c7d8e9-0415-4d26-8f70-8192a3b4c5d6", - "StartTime": "2026-06-06T06:00:00.170Z", - "EndTime": "2026-06-06T06:00:35.480Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "c7d8e9f0-1526-4d37-9081-92a3b4c5d6e7", - "Info": "Send SMTP Mail Message: Failure sending mail.\r\n\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host", - "CreationTime": "2026-06-06T05:59:59.900Z", - "ReleaseName": "WeeklyReportDispatcher", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6061407, - "OrganizationUnitId": 7791618, - "FolderKey": "e5f6a7b8-9304-4d15-8e6f-708192a3b4c5", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-06T06:00:35.480Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b6c7d8e9-0415-4d26-8f70-8192a3b4c5d6", - "Id": 682712006 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/75949d659e.json deleted file mode 100644 index af4efcf932..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/75949d659e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b6c7d8e9-0415-4d26-8f70-8192a3b4c5d6", - "State": "Faulted", - "ProcessName": "WeeklyReportDispatcher", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-06T06:00:00.170Z", - "EndTime": "2026-06-06T06:00:35.480Z", - "CreationTime": "2026-06-06T05:59:59.900Z" - } - ], - "Pagination": { "Returned": 1, "Limit": 50, "Offset": 0, "HasMore": false } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/README.md deleted file mode 100644 index 616ee79cbb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Send SMTP Mail Message — Port / Transport-Security Mismatch (Branch 3) - -Runtime troubleshooting scenario for `UiPath.Mail.SMTP.Activities` `Send SMTP Mail Message` -(`SmtpSendMail`), covering the **connection / port-vs-`SecureConnection` mismatch** of -`send-smtp-mail-failures.md` **Branch 3**. - -## What this scenario exercises - -An unattended job hangs ~35 s then faults with `System.Net.Mail.SmtpException: Failure sending mail` -wrapping an `IOException` → `SocketException` ("An existing connection was forcibly closed"). The -activity pairs `Port=465` with `SecureConnection=StartTls`, but 465 expects **implicit** SSL — so the -STARTTLS negotiation fails and the server drops the socket. The agent must attribute the failure to the -Port↔`SecureConnection` mismatch (not auth, not relay, not a server outage) and prescribe the correct -pairing: **587 + StartTls** or **465 + SSL**. - -## Discriminator - -- The socket **connected** (`TCP connected`) and the failure is at the **TLS negotiation** — so it is - not a server-unreachable outage. -- There is **no `535`/`5.7.x` auth code** and **no `550`/`5.7.1` relay code** — so it is not Branch 1/2 - (auth) or Branch 4/5 (relay/recipient). The error is a `SocketException` at transport setup. -- Source shows the decisive mispairing: `Port="465"` + `SecureConnection="StartTls"`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `SmtpSendMail` with `Server="smtp.office365.com"`, `Port="465"`, `SecureConnection="StartTls"` | -| `data/m/r/` | synthetic canned `uip` responses; the smoking gun is `job-logs.json` (connect 465 StartTls → STARTTLS gets no response on an implicit-TLS port → connection dropped → `SmtpException`/`SocketException`); `docsai ask` passthrough | - -Diagnosis is not leaked in any agent-visible name (project `WeeklyReportDispatcher`, activity "Send SMTP -Mail Message"); the prompt states only the observed symptom (job hangs then faults with the pasted -`Failure sending mail` / socket error). - -## Success criteria - -`skill_triggered` + `llm_judge` against `RESOLUTION.md`: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the Port↔`SecureConnection` mismatch (465 + StartTls) as the cause and prescribed the - correct pairing (587 + StartTls, or 465 + SSL) — not an auth, relay, or server-outage misdiagnosis, - and not disabling `SecureConnection`. - -Playbook: `references/activity-packages/mail-activities/playbooks/send-smtp-mail-failures.md` (Branch 3). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/b8d18266dc.json deleted file mode 100644 index 6e84a48598..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { "Level": "Info", "TimeStamp": "2026-06-06T06:00:00.210Z", "Message": "Execution started" }, - { "Level": "Trace", "TimeStamp": "2026-06-06T06:00:01.020Z", "Message": "[Send SMTP Mail Message] Opening connection to smtp.office365.com:465 (SecureConnection=StartTls)" }, - { "Level": "Trace", "TimeStamp": "2026-06-06T06:00:01.640Z", "Message": "[Send SMTP Mail Message] TCP connected; sending EHLO then STARTTLS command on port 465" }, - { "Level": "Warn", "TimeStamp": "2026-06-06T06:00:16.220Z", "Message": "[Send SMTP Mail Message] No SMTP greeting/response to STARTTLS on port 465; the server appears to expect implicit TLS on this port. Retrying/holding the socket" }, - { "Level": "Error", "TimeStamp": "2026-06-06T06:00:35.360Z", "Message": "[Send SMTP Mail Message] Transport connection dropped during TLS negotiation on smtp.office365.com:465" }, - { "Level": "Error", "TimeStamp": "2026-06-06T06:00:35.420Z", "Message": "Send SMTP Mail Message: Failure sending mail.\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host" }, - { "Level": "Info", "TimeStamp": "2026-06-06T06:00:35.470Z", "Message": "Execution ended in: 00:00:35" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/d742317695.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/d742317695.json deleted file mode 100644 index 065f39c1c0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/d742317695.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { "Level": "Error", "TimeStamp": "2026-06-06T06:00:35.360Z", "Message": "[Send SMTP Mail Message] Transport connection dropped during TLS negotiation on smtp.office365.com:465" }, - { "Level": "Error", "TimeStamp": "2026-06-06T06:00:35.420Z", "Message": "Send SMTP Mail Message: Failure sending mail.\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/ee2628e6b6.json deleted file mode 100644 index af0a9ef311..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e5f6a7b8-9304-4d15-8e6f-708192a3b4c5", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/manifest.json deleted file mode 100644 index 59791d8d8e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send SMTP Mail Message port/transport-security mismatch sub-case (Branch 3 of send-smtp-mail-failures); mock dispatcher only. The activity targets port 465 with SecureConnection=StartTls; 465 is implicit SSL, so the STARTTLS negotiation fails and the connection is dropped (SmtpException wrapping a socket/handshake error). docsai proxied; --level Error rule precedes the generic logs rule. Token-aware matching, first match wins.", - "rules": [ - { "match": "docsai ask", "passthrough": true }, - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs logs --level Error", "file": "d742317695.json" }, - { "match": "or jobs logs", "file": "b8d18266dc.json" }, - { "match": "or jobs get", "file": "3db6b96527.json" } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0, "_doc": "Permissive fallback for unmatched commands." } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..9c8d443bc9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-06T06:00:35.360Z\", \"Message\": \"[Send SMTP Mail Message] Transport connection dropped during TLS negotiation on smtp.office365.com:465\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-06T06:00:35.420Z\", \"Message\": \"Send SMTP Mail Message: Failure sending mail.\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\" }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e5f6a7b8-9304-4d15-8e6f-708192a3b4c5\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b6c7d8e9-0415-4d26-8f70-8192a3b4c5d6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"WeeklyReportDispatcher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-06T06:00:00.170Z\",\r\n \"EndTime\": \"2026-06-06T06:00:35.480Z\",\r\n \"CreationTime\": \"2026-06-06T05:59:59.900Z\"\r\n }\r\n ],\r\n \"Pagination\": { \"Returned\": 1, \"Limit\": 50, \"Offset\": 0, \"HasMore\": false }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n { \"Level\": \"Info\", \"TimeStamp\": \"2026-06-06T06:00:00.210Z\", \"Message\": \"Execution started\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-06T06:00:01.020Z\", \"Message\": \"[Send SMTP Mail Message] Opening connection to smtp.office365.com:465 (SecureConnection=StartTls)\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-06T06:00:01.640Z\", \"Message\": \"[Send SMTP Mail Message] TCP connected; sending EHLO then STARTTLS command on port 465\" },\r\n { \"Level\": \"Warn\", \"TimeStamp\": \"2026-06-06T06:00:16.220Z\", \"Message\": \"[Send SMTP Mail Message] No SMTP greeting/response to STARTTLS on port 465; the server appears to expect implicit TLS on this port. Retrying/holding the socket\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-06T06:00:35.360Z\", \"Message\": \"[Send SMTP Mail Message] Transport connection dropped during TLS negotiation on smtp.office365.com:465\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-06T06:00:35.420Z\", \"Message\": \"Send SMTP Mail Message: Failure sending mail.\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\" },\r\n { \"Level\": \"Info\", \"TimeStamp\": \"2026-06-06T06:00:35.470Z\", \"Message\": \"Execution ended in: 00:00:35\" }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b6c7d8e9-0415-4d26-8f70-8192a3b4c5d6\",\r\n \"StartTime\": \"2026-06-06T06:00:00.170Z\",\r\n \"EndTime\": \"2026-06-06T06:00:35.480Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"c7d8e9f0-1526-4d37-9081-92a3b4c5d6e7\",\r\n \"Info\": \"Send SMTP Mail Message: Failure sending mail.\\r\\n\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: Failure sending mail. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\",\r\n \"CreationTime\": \"2026-06-06T05:59:59.900Z\",\r\n \"ReleaseName\": \"WeeklyReportDispatcher\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6061407,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"e5f6a7b8-9304-4d15-8e6f-708192a3b4c5\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-06T06:00:35.480Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b6c7d8e9-0415-4d26-8f70-8192a3b4c5d6\",\r\n \"Id\": 682712006\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/task.yaml index b65897ca9a..cdb026d4e5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-port-tls-mismatch/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/README.md new file mode 100644 index 0000000000..c3a2c821a7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/README.md @@ -0,0 +1,51 @@ +# Send SMTP Mail Message — Relay / Sender Not Permitted (Branch 4) + +Runtime troubleshooting scenario for `UiPath.Mail.SMTP.Activities` `Send SMTP Mail Message` +(`SmtpSendMail`), covering the **relay / sender-not-permitted** sub-case of +`send-smtp-mail-failures.md` **Branch 4**. + +## What this scenario exercises + +An unattended job faults with `System.Net.Mail.SmtpException: Mailbox unavailable. The server response +was: 550 5.7.60 ... Client does not have permissions to send as this sender.` Connection, STARTTLS, and +authentication all succeed as `svc-billing@contoso-test.com`, but the activity's `From` is +`noreply@marketing-partner.com` — an address the authenticated account may not send as — so the server +rejects the message at the envelope step. The agent must attribute the failure to the send-as / relay +policy (authenticated identity ≠ `From`) and prescribe aligning `From` with the authenticated account / +a granted Send-As alias (or relaying via an authorized connector) — NOT an auth or transport fix. + +## Discriminator + +- Login **succeeded** (`authenticated as svc-billing@contoso-test.com`) — so it is not Branch 1/2 + (auth); there is no `535`/`5.7.3`/`5.7.139`. +- STARTTLS negotiated on 587 and the session reached `MAIL FROM`/`RCPT TO` — so it is not Branch 3 + (connection/TLS). +- The rejection code is `5.7.60` (sender permissions), not `5.1.1` (bad recipient) or `552` (size) — so + it is not Branch 5. +- Source shows the decisive mismatch: `Email="svc-billing@contoso-test.com"` (auth) vs + `From="noreply@marketing-partner.com"`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `SmtpSendMail` authenticating as `svc-billing@contoso-test.com` with `From="noreply@marketing-partner.com"` | + +Diagnosis is not disclosed in any agent-visible name (project `CustomerAlertSender`, activity "Send SMTP +Mail Message"); the prompt states only the observed symptom (job faults with the pasted `550 5.7.60` +error). + +## Success criteria + +`skill_triggered` + `llm_judge` against `RESOLUTION.md`: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the send-as / relay rejection (authenticated identity ≠ `From`) as the cause and + prescribed aligning `From` with the authenticated account / a granted Send-As alias (or an authorized + relay) — not an auth-credential, transport, or recipient misdiagnosis. + +Playbook: `references/activity-packages/mail-activities/playbooks/send-smtp-mail-failures.md` (Branch 4). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/3db6b96527.json deleted file mode 100644 index 3a63564bec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/3db6b96527.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c7d8e9f0-1526-4f37-8081-92a3b4c5d6e8", - "StartTime": "2026-06-07T04:30:00.180Z", - "EndTime": "2026-06-07T04:30:04.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "d8e9f0a1-2637-4f48-9192-a3b4c5d6e7f9", - "Info": "Send SMTP Mail Message: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\r\n\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.", - "CreationTime": "2026-06-07T04:29:59.900Z", - "ReleaseName": "CustomerAlertSender", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6071407, - "OrganizationUnitId": 7791618, - "FolderKey": "f6a7b8c9-0415-4e26-9f70-8192a3b4c5d7", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-07T04:30:04.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c7d8e9f0-1526-4f37-8081-92a3b4c5d6e8", - "Id": 682812017 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/75949d659e.json deleted file mode 100644 index 26089535cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/75949d659e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c7d8e9f0-1526-4f37-8081-92a3b4c5d6e8", - "State": "Faulted", - "ProcessName": "CustomerAlertSender", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-07T04:30:00.180Z", - "EndTime": "2026-06-07T04:30:04.220Z", - "CreationTime": "2026-06-07T04:29:59.900Z" - } - ], - "Pagination": { "Returned": 1, "Limit": 50, "Offset": 0, "HasMore": false } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/README.md deleted file mode 100644 index efd6a54e53..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# Send SMTP Mail Message — Relay / Sender Not Permitted (Branch 4) - -Runtime troubleshooting scenario for `UiPath.Mail.SMTP.Activities` `Send SMTP Mail Message` -(`SmtpSendMail`), covering the **relay / sender-not-permitted** sub-case of -`send-smtp-mail-failures.md` **Branch 4**. - -## What this scenario exercises - -An unattended job faults with `System.Net.Mail.SmtpException: Mailbox unavailable. The server response -was: 550 5.7.60 ... Client does not have permissions to send as this sender.` Connection, STARTTLS, and -authentication all succeed as `svc-billing@contoso-test.com`, but the activity's `From` is -`noreply@marketing-partner.com` — an address the authenticated account may not send as — so the server -rejects the message at the envelope step. The agent must attribute the failure to the send-as / relay -policy (authenticated identity ≠ `From`) and prescribe aligning `From` with the authenticated account / -a granted Send-As alias (or relaying via an authorized connector) — NOT an auth or transport fix. - -## Discriminator - -- Login **succeeded** (`authenticated as svc-billing@contoso-test.com`) — so it is not Branch 1/2 - (auth); there is no `535`/`5.7.3`/`5.7.139`. -- STARTTLS negotiated on 587 and the session reached `MAIL FROM`/`RCPT TO` — so it is not Branch 3 - (connection/TLS). -- The rejection code is `5.7.60` (sender permissions), not `5.1.1` (bad recipient) or `552` (size) — so - it is not Branch 5. -- Source shows the decisive mismatch: `Email="svc-billing@contoso-test.com"` (auth) vs - `From="noreply@marketing-partner.com"`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `SmtpSendMail` authenticating as `svc-billing@contoso-test.com` with `From="noreply@marketing-partner.com"` | -| `data/m/r/` | synthetic canned `uip` responses; the smoking gun is `job-logs.json` (auth OK → `MAIL FROM:` → `550 5.7.60 ... send as this sender`); `docsai ask` passthrough | - -Diagnosis is not leaked in any agent-visible name (project `CustomerAlertSender`, activity "Send SMTP -Mail Message"); the prompt states only the observed symptom (job faults with the pasted `550 5.7.60` -error). - -## Success criteria - -`skill_triggered` + `llm_judge` against `RESOLUTION.md`: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the send-as / relay rejection (authenticated identity ≠ `From`) as the cause and - prescribed aligning `From` with the authenticated account / a granted Send-As alias (or an authorized - relay) — not an auth-credential, transport, or recipient misdiagnosis. - -Playbook: `references/activity-packages/mail-activities/playbooks/send-smtp-mail-failures.md` (Branch 4). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/b8d18266dc.json deleted file mode 100644 index 46568478c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { "Level": "Info", "TimeStamp": "2026-06-07T04:30:00.220Z", "Message": "Execution started" }, - { "Level": "Trace", "TimeStamp": "2026-06-07T04:30:01.020Z", "Message": "[Send SMTP Mail Message] Connecting to smtp.office365.com:587 (SecureConnection=StartTls)" }, - { "Level": "Trace", "TimeStamp": "2026-06-07T04:30:01.760Z", "Message": "[Send SMTP Mail Message] STARTTLS negotiated; authenticated as svc-billing@contoso-test.com" }, - { "Level": "Trace", "TimeStamp": "2026-06-07T04:30:02.410Z", "Message": "[Send SMTP Mail Message] MAIL FROM:; RCPT TO:" }, - { "Level": "Error", "TimeStamp": "2026-06-07T04:30:03.980Z", "Message": "[Send SMTP Mail Message] Server rejected the message: 550 5.7.60 SMTP; Client does not have permissions to send as this sender (authenticated as svc-billing@contoso-test.com, From noreply@marketing-partner.com)" }, - { "Level": "Error", "TimeStamp": "2026-06-07T04:30:04.060Z", "Message": "Send SMTP Mail Message: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender." }, - { "Level": "Info", "TimeStamp": "2026-06-07T04:30:04.200Z", "Message": "Execution ended in: 00:00:04" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/d742317695.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/d742317695.json deleted file mode 100644 index 1d5cde044a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/d742317695.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { "Level": "Error", "TimeStamp": "2026-06-07T04:30:03.980Z", "Message": "[Send SMTP Mail Message] Server rejected the message: 550 5.7.60 SMTP; Client does not have permissions to send as this sender (authenticated as svc-billing@contoso-test.com, From noreply@marketing-partner.com)" }, - { "Level": "Error", "TimeStamp": "2026-06-07T04:30:04.060Z", "Message": "Send SMTP Mail Message: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\r\nin Main.xaml\r\n at SmtpSendMail \"Send SMTP Mail Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Net.Mail.SmtpException: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender." } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/ee2628e6b6.json deleted file mode 100644 index fd76c24bc5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f6a7b8c9-0415-4e26-9f70-8192a3b4c5d7", - "Name": "RPA Production", - "Path": "RPA Production", - "Description": "Production folder for the finance RPA portfolio.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/manifest.json deleted file mode 100644 index 904864b9cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/m/r/manifest.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Send SMTP Mail Message relay / sender-not-permitted sub-case (Branch 4 of send-smtp-mail-failures); mock dispatcher only. Authentication as svc-billing@contoso-test.com succeeds, but the From address (noreply@marketing-partner.com) is not one the authenticated account may send as, so the server rejects with 550 5.7.60. docsai proxied; --level Error rule precedes the generic logs rule. Token-aware matching, first match wins.", - "rules": [ - { "match": "docsai ask", "passthrough": true }, - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs logs --level Error", "file": "d742317695.json" }, - { "match": "or jobs logs", "file": "b8d18266dc.json" }, - { "match": "or jobs get", "file": "3db6b96527.json" } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0, "_doc": "Permissive fallback for unmatched commands." } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/uip-fixture.json new file mode 100644 index 0000000000..03323a2976 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-07T04:30:03.980Z\", \"Message\": \"[Send SMTP Mail Message] Server rejected the message: 550 5.7.60 SMTP; Client does not have permissions to send as this sender (authenticated as svc-billing@contoso-test.com, From noreply@marketing-partner.com)\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-07T04:30:04.060Z\", \"Message\": \"Send SMTP Mail Message: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\" }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f6a7b8c9-0415-4e26-9f70-8192a3b4c5d7\",\r\n \"Name\": \"RPA Production\",\r\n \"Path\": \"RPA Production\",\r\n \"Description\": \"Production folder for the finance RPA portfolio.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-5e6f-4071-8293-a4b5c6d7e8f9\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c7d8e9f0-1526-4f37-8081-92a3b4c5d6e8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerAlertSender\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-07T04:30:00.180Z\",\r\n \"EndTime\": \"2026-06-07T04:30:04.220Z\",\r\n \"CreationTime\": \"2026-06-07T04:29:59.900Z\"\r\n }\r\n ],\r\n \"Pagination\": { \"Returned\": 1, \"Limit\": 50, \"Offset\": 0, \"HasMore\": false }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n { \"Level\": \"Info\", \"TimeStamp\": \"2026-06-07T04:30:00.220Z\", \"Message\": \"Execution started\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-07T04:30:01.020Z\", \"Message\": \"[Send SMTP Mail Message] Connecting to smtp.office365.com:587 (SecureConnection=StartTls)\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-07T04:30:01.760Z\", \"Message\": \"[Send SMTP Mail Message] STARTTLS negotiated; authenticated as svc-billing@contoso-test.com\" },\r\n { \"Level\": \"Trace\", \"TimeStamp\": \"2026-06-07T04:30:02.410Z\", \"Message\": \"[Send SMTP Mail Message] MAIL FROM:; RCPT TO:\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-07T04:30:03.980Z\", \"Message\": \"[Send SMTP Mail Message] Server rejected the message: 550 5.7.60 SMTP; Client does not have permissions to send as this sender (authenticated as svc-billing@contoso-test.com, From noreply@marketing-partner.com)\" },\r\n { \"Level\": \"Error\", \"TimeStamp\": \"2026-06-07T04:30:04.060Z\", \"Message\": \"Send SMTP Mail Message: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\" },\r\n { \"Level\": \"Info\", \"TimeStamp\": \"2026-06-07T04:30:04.200Z\", \"Message\": \"Execution ended in: 00:00:04\" }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c7d8e9f0-1526-4f37-8081-92a3b4c5d6e8\",\r\n \"StartTime\": \"2026-06-07T04:30:00.180Z\",\r\n \"EndTime\": \"2026-06-07T04:30:04.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"d8e9f0a1-2637-4f48-9192-a3b4c5d6e7f9\",\r\n \"Info\": \"Send SMTP Mail Message: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\\r\\n\\r\\nin Main.xaml\\r\\n at SmtpSendMail \\\"Send SMTP Mail Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Net.Mail.SmtpException: Mailbox unavailable. The server response was: 550 5.7.60 SMTP; Client does not have permissions to send as this sender.\",\r\n \"CreationTime\": \"2026-06-07T04:29:59.900Z\",\r\n \"ReleaseName\": \"CustomerAlertSender\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6071407,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"f6a7b8c9-0415-4e26-9f70-8192a3b4c5d7\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-07T04:30:04.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c7d8e9f0-1526-4f37-8081-92a3b4c5d6e8\",\r\n \"Id\": 682812017\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/task.yaml index 9c46dcfe5c..6fa1465d42 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/mail-send-smtp-relay-denied/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: node: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/README.md new file mode 100644 index 0000000000..12fd7e6756 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/README.md @@ -0,0 +1,36 @@ +# O365 App Scope Asset Not Found — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The Microsoft 365 Scope activity in process "ERN" is configured to read its connection from an Orchestrator asset, and that asset does not resolve when the job runs in folder "Shared". The asset identifier comes back null, so connection resolution throws before any Microsoft 365 (Graph) call is ever made. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name o365-app-scope-asset-not-found --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/1e3011fc7d.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/1e3011fc7d.json deleted file mode 100644 index ec22d32d0b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/1e3011fc7d.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "Info": "Could not retrieve the selected asset: 'Value cannot be null. (Parameter 'assetId')'\r\n\r\nin Main.xaml\r\n at Office365ApplicationScope \"Microsoft 365 Scope (Orchestrator asset that does not resolve)\"\r\n at Sequence \"O365 Application Scope / connection misconfiguration repro\"\r\n at Main \"Main\"\r\n\r\nUiPath.MicrosoftOffice365.Office365Exception: Could not retrieve the selected asset: 'Value cannot be null. (Parameter 'assetId')'\r\n ---> System.ArgumentNullException: Value cannot be null. (Parameter 'assetId')\r\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\r\n --- End of inner exception stack trace ---\r\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.GetConnectionAsync(IOrchestratorClient orchestratorClient, IMemoryCache cache, ActivityConfigLocation configLocation, Boolean useCache, String account, Func`1 assetIdFunc, Func`1 assetNameFunc, Func`1 robotKeyFunc, Func`1 fullPathFunc, Func`1 manualFullPathFunc, Func`1 inputModeFunc, AuthenticationType authType, Func`1 clientIdFunc, Func`1 tenantIdFunc, Func`1 clientSecretFunc, Func`1 secureClientSecretFunc, Func`1 emailAddressFunc, Func`1 impersonatedUserEmailAddressFunc, HostingEnvironment env, List`1 interactiveScopes, Func`1 passwordFunc, Func`1 securePasswordFunc, Func`1 certificateAsBase64Func, Func`1 certificatePasswordFunc, Func`1 usernameFunc, MasterDataStoreLocation dataStore, Func`1 folderFunc, String connector, String connectionAccountName, String connectionId, Int32 timeoutMs, Nullable`1 useBroker, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-09T12:31:11.730Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53466, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-09T12:31:13.000Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "292c80e4bc4f41e398970fe73e0c64db", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 4015052 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/5c090b4bdc.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/5c090b4bdc.json deleted file mode 100644 index 32cb2873c8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/5c090b4bdc.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-b77a-5b4e61e7fe60", - "TraceId": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "ParentId": null, - "Name": "Robot Job 292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "StartTime": "2026-06-09T12:31:12.0915344Z", - "EndTime": "2026-06-09T12:31:12.3536927Z", - "Attributes": "{\"Exception\":\"UiPath.MicrosoftOffice365.Office365Exception: Could not retrieve the selected asset: 'Value cannot be null. (Parameter 'assetId')'\\r\\n ---> System.ArgumentNullException: Value cannot be null. (Parameter 'assetId')\\r\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.GetConnectionAsync(IOrchestratorClient orchestratorClient, IMemoryCache cache, ActivityConfigLocation configLocation, Boolean useCache, String account, Func`1 assetIdFunc, Func`1 assetNameFunc, Func`1 robotKeyFunc, Func`1 fullPathFunc, Func`1 manualFullPathFunc, Func`1 inputModeFunc, AuthenticationType authType, Func`1 clientIdFunc, Func`1 tenantIdFunc, Func`1 clientSecretFunc, Func`1 secureClientSecretFunc, Func`1 emailAddressFunc, Func`1 impersonatedUserEmailAddressFunc, HostingEnvironment env, List`1 interactiveScopes, Func`1 passwordFunc, Func`1 securePasswordFunc, Func`1 certificateAsBase64Func, Func`1 certificatePasswordFunc, Func`1 usernameFunc, MasterDataStoreLocation dataStore, Func`1 folderFunc, String connector, String connectionAccountName, String connectionId, Int32 timeoutMs, Nullable`1 useBroker, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "b16092a1-0c68-436c-bf3a-147ab53b6762", - "JobKey": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-09T12:31:12.7157721Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-0399-e10338135d8a", - "TraceId": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "ParentId": "00000000-0000-0000-b77a-5b4e61e7fe60", - "Name": "Microsoft 365 Scope (Orchestrator asset that does not resolve)", - "StartTime": "2026-06-09T12:31:12.1787193Z", - "EndTime": "2026-06-09T12:31:12.2136524Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"OrchestratorAsset\",\"Private.BrowserItemId\":\"999999\",\"Private.BrowserItemFriendlyName\":\"MissingO365Asset\",\"Private.BrowserItemFullPath\":\"uipath.missing.o365.asset\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope (Orchestrator asset that does not resolve)\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "b16092a1-0c68-436c-bf3a-147ab53b6762", - "JobKey": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-09T12:31:12.7157721Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/README.md deleted file mode 100644 index d9ad03c347..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# O365 App Scope Asset Not Found — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The Microsoft 365 Scope activity in process "ERN" is configured to read its connection from an Orchestrator asset, and that asset does not resolve when the job runs in folder "Shared". The asset identifier comes back null, so connection resolution throws before any Microsoft 365 (Graph) call is ever made. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name o365-app-scope-asset-not-found --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/d9a8fa27ff.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/d9a8fa27ff.json deleted file mode 100644 index 46bdd7fd51..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/d9a8fa27ff.json +++ /dev/null @@ -1,320 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 22, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/manifest.json deleted file mode 100644 index b06c844780..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py. Adjusted: docsai -> passthrough (per tests CLAUDE.md); the original session used a $JK shell variable for the job key, so the recorded 'or jobs get \"$JK\"' / 'or jobs traces \"$JK\"' rules were rewritten to the literal key 292c80e4-... with quotes/trailing flags dropped, so a fresh agent's commands match the substring dispatcher.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "d9a8fa27ff.json" - }, - { - "match": "or jobs get 292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "file": "1e3011fc7d.json" - }, - { - "match": "or jobs traces 292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "file": "5c090b4bdc.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by rules returns an empty array." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..395ad93bbb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/data/uip-fixture.json @@ -0,0 +1,53 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 22,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "292c80e4-bc4f-41e3-9897-0fe73e0c64db" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"Info\": \"Could not retrieve the selected asset: 'Value cannot be null. (Parameter 'assetId')'\\r\\n\\r\\nin Main.xaml\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope (Orchestrator asset that does not resolve)\\\"\\r\\n at Sequence \\\"O365 Application Scope / connection misconfiguration repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.MicrosoftOffice365.Office365Exception: Could not retrieve the selected asset: 'Value cannot be null. (Parameter 'assetId')'\\r\\n ---> System.ArgumentNullException: Value cannot be null. (Parameter 'assetId')\\r\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.GetConnectionAsync(IOrchestratorClient orchestratorClient, IMemoryCache cache, ActivityConfigLocation configLocation, Boolean useCache, String account, Func`1 assetIdFunc, Func`1 assetNameFunc, Func`1 robotKeyFunc, Func`1 fullPathFunc, Func`1 manualFullPathFunc, Func`1 inputModeFunc, AuthenticationType authType, Func`1 clientIdFunc, Func`1 tenantIdFunc, Func`1 clientSecretFunc, Func`1 secureClientSecretFunc, Func`1 emailAddressFunc, Func`1 impersonatedUserEmailAddressFunc, HostingEnvironment env, List`1 interactiveScopes, Func`1 passwordFunc, Func`1 securePasswordFunc, Func`1 certificateAsBase64Func, Func`1 certificatePasswordFunc, Func`1 usernameFunc, MasterDataStoreLocation dataStore, Func`1 folderFunc, String connector, String connectionAccountName, String connectionId, Int32 timeoutMs, Nullable`1 useBroker, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53466,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"292c80e4bc4f41e398970fe73e0c64db\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 4015052\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "292c80e4-bc4f-41e3-9897-0fe73e0c64db" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-b77a-5b4e61e7fe60\",\r\n \"TraceId\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"StartTime\": \"2026-06-09T12:31:12.0915344Z\",\r\n \"EndTime\": \"2026-06-09T12:31:12.3536927Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.MicrosoftOffice365.Office365Exception: Could not retrieve the selected asset: 'Value cannot be null. (Parameter 'assetId')'\\\\r\\\\n ---> System.ArgumentNullException: Value cannot be null. (Parameter 'assetId')\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.<>c__DisplayClass3_0.<g__GetApplicationFromAssetAsync|0>d.MoveNext()\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Utilities.Office365ApplicationScopeUtils.GetConnectionAsync(IOrchestratorClient orchestratorClient, IMemoryCache cache, ActivityConfigLocation configLocation, Boolean useCache, String account, Func`1 assetIdFunc, Func`1 assetNameFunc, Func`1 robotKeyFunc, Func`1 fullPathFunc, Func`1 manualFullPathFunc, Func`1 inputModeFunc, AuthenticationType authType, Func`1 clientIdFunc, Func`1 tenantIdFunc, Func`1 clientSecretFunc, Func`1 secureClientSecretFunc, Func`1 emailAddressFunc, Func`1 impersonatedUserEmailAddressFunc, HostingEnvironment env, List`1 interactiveScopes, Func`1 passwordFunc, Func`1 securePasswordFunc, Func`1 certificateAsBase64Func, Func`1 certificatePasswordFunc, Func`1 usernameFunc, MasterDataStoreLocation dataStore, Func`1 folderFunc, String connector, String connectionAccountName, String connectionId, Int32 timeoutMs, Nullable`1 useBroker, CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"b16092a1-0c68-436c-bf3a-147ab53b6762\",\r\n \"JobKey\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-09T12:31:12.7157721Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-0399-e10338135d8a\",\r\n \"TraceId\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"ParentId\": \"00000000-0000-0000-b77a-5b4e61e7fe60\",\r\n \"Name\": \"Microsoft 365 Scope (Orchestrator asset that does not resolve)\",\r\n \"StartTime\": \"2026-06-09T12:31:12.1787193Z\",\r\n \"EndTime\": \"2026-06-09T12:31:12.2136524Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"OrchestratorAsset\\\",\\\"Private.BrowserItemId\\\":\\\"999999\\\",\\\"Private.BrowserItemFriendlyName\\\":\\\"MissingO365Asset\\\",\\\"Private.BrowserItemFullPath\\\":\\\"uipath.missing.o365.asset\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope (Orchestrator asset that does not resolve)\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"b16092a1-0c68-436c-bf3a-147ab53b6762\",\r\n \"JobKey\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-09T12:31:12.7157721Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/task.yaml index 1b3a28d431..142e2311ef 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-app-scope-asset-not-found/task.yaml @@ -18,21 +18,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/README.md new file mode 100644 index 0000000000..5228e59ef3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/README.md @@ -0,0 +1,46 @@ +# O365 Copy Item Null DriveItem — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `DriveItemCopier` (key `909ed39c-08a1-48fb-921a-06d332f8f5fb`), +faulted with a raw `System.ArgumentNullException: Value cannot be null. +(Parameter 'DriveItem')` thrown by the legacy O365 `CopyItem` activity. Job +trace spans proved the binding chain: an upstream `FindFilesAndFolders` +(query `no-such-file-zzz-repro`, `First` output bound to `foundItem`) matched +nothing and left `foundItem` null; `CopyItem`'s `DriveItem` input is bound to +that same variable. Fix: correct the search criteria and/or add a null guard +between the search and the copy. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job traces carried the variable-binding chain). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +If the source investigation changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-copyitem-null-driveitem --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/06d7ede76c.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/06d7ede76c.json deleted file mode 100644 index 3fb6506713..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/06d7ede76c.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T17:12:13.804Z", - "Message": "Copy Item: Value cannot be null. (Parameter 'DriveItem')\r\nin Main.xaml\r\n at CopyItem \"Copy Item\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Copy Item null DriveItem repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentNullException\r\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/2519cdb8ee.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/2519cdb8ee.json deleted file mode 100644 index 1dd18bf65e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/2519cdb8ee.json +++ /dev/null @@ -1,348 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 24, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/5f88713a10.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/5f88713a10.json deleted file mode 100644 index 24779c469b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/5f88713a10.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "14bcebd6-6eeb-4933-ade8-947658d27b70", - "Info": "Value cannot be null. (Parameter 'DriveItem')\r\n\r\nin Main.xaml\r\n at CopyItem \"Copy Item\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Copy Item null DriveItem repro\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'DriveItem')\r\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T16:53:17.843Z", - "ReleaseName": "DriveItemCopier", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53467, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T17:12:14.247Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "909ed39c08a148fb921a06d332f8f5fb", - "Id": 4015106 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/7ef81ffd7e.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/7ef81ffd7e.json deleted file mode 100644 index a9f9069ce6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/7ef81ffd7e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T17:12:14.253Z", - "JobId": 4015106 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T17:12:06.217Z", - "JobId": 4015106 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T16:53:17.913Z", - "JobId": 4015106 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/README.md deleted file mode 100644 index 6754dd26f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# O365 Copy Item Null DriveItem — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `DriveItemCopier` (key `909ed39c-08a1-48fb-921a-06d332f8f5fb`), -faulted with a raw `System.ArgumentNullException: Value cannot be null. -(Parameter 'DriveItem')` thrown by the legacy O365 `CopyItem` activity. Job -trace spans proved the binding chain: an upstream `FindFilesAndFolders` -(query `no-such-file-zzz-repro`, `First` output bound to `foundItem`) matched -nothing and left `foundItem` null; `CopyItem`'s `DriveItem` input is bound to -that same variable. Fix: correct the search criteria and/or add a null guard -between the search and the copy. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job traces carried the variable-binding chain). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -If the source investigation changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-copyitem-null-driveitem --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/bc882ab324.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/bc882ab324.json deleted file mode 100644 index cc1059ad0c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/bc882ab324.json +++ /dev/null @@ -1,110 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-8776-0214877dc4e4", - "TraceId": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ParentId": null, - "Name": "Robot Job 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "StartTime": "2026-06-10T17:12:10.3374556Z", - "EndTime": "2026-06-10T17:12:13.8319517Z", - "Attributes": "{\"Exception\":\"System.ArgumentNullException: Value cannot be null. (Parameter 'DriveItem')\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "ec2c2000-846e-4020-ba3f-297f3dfce24e", - "JobKey": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:12:14.0905943Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-a1c2-a0228f5e60cb", - "TraceId": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ParentId": "00000000-0000-0000-8776-0214877dc4e4", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-10T17:12:10.4261792Z", - "EndTime": "2026-06-10T17:12:10.4653291Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"PropertiesPanel\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "ec2c2000-846e-4020-ba3f-297f3dfce24e", - "JobKey": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:12:11.0505953Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-9ff8-913f61a075e5", - "TraceId": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ParentId": "00000000-0000-0000-8776-0214877dc4e4", - "Name": "Find Files And Folders", - "StartTime": "2026-06-10T17:12:10.6087938Z", - "EndTime": "2026-06-10T17:12:10.6096896Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Files.FindFilesAndFolders\",\"Private.Query\":\"no-such-file-zzz-repro\",\"Private.First\":\"foundItem\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Find Files And Folders\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "ec2c2000-846e-4020-ba3f-297f3dfce24e", - "JobKey": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:12:11.0505953Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-a1c9-53ef7ebd7f3a", - "TraceId": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ParentId": "00000000-0000-0000-8776-0214877dc4e4", - "Name": "Copy Item", - "StartTime": "2026-06-10T17:12:13.7782756Z", - "EndTime": "2026-06-10T17:12:13.7790144Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Files.CopyItem\",\"Private.DriveItem\":\"foundItem\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Copy Item\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "ec2c2000-846e-4020-ba3f-297f3dfce24e", - "JobKey": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:12:14.0905943Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/manifest.json deleted file mode 100644 index 3370ec92f6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/m/r/manifest.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "file": "5f88713a10.json" - }, - { - "match": "or jobs logs 909ed39c-08a1-48fb-921a-06d332f8f5fb --level Error", - "file": "06d7ede76c.json" - }, - { - "match": "or jobs logs 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "file": "06d7ede76c.json" - }, - { - "match": "or jobs history 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "file": "7ef81ffd7e.json" - }, - { - "match": "or jobs traces 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "file": "bc882ab324.json" - }, - { - "match": "traces spans get --job-key 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "file": "bc882ab324.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same span payload as `or jobs traces` — without this rule the call fell through to the empty unmocked_default and the agent concluded no upstream activity ran (observed degraded replicate, judge 0.6)." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "2519cdb8ee.json" - }, - { - "match": "or jobs list", - "file": "2519cdb8ee.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "min": 1, - "description": "Agent must fetch the failing job to read the raw ArgumentNullException (Parameter 'DriveItem')." - }, - { - "pattern": "traces", - "min": 1, - "description": "Trace spans carry the variable-binding chain (FindFilesAndFolders First='foundItem' -> CopyItem DriveItem='foundItem'); the agent may retrieve them via `or jobs traces ` or `traces spans get --job-key `." - }, - { - "pattern": "or jobs logs 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history 909ed39c-08a1-48fb-921a-06d332f8f5fb", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/uip-fixture.json new file mode 100644 index 0000000000..3ad4059f33 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/data/uip-fixture.json @@ -0,0 +1,142 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "909ed39c-08a1-48fb-921a-06d332f8f5fb", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T17:12:13.804Z\",\r\n \"Message\": \"Copy Item: Value cannot be null. (Parameter 'DriveItem')\\r\\nin Main.xaml\\r\\n at CopyItem \\\"Copy Item\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Copy Item null DriveItem repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "909ed39c-08a1-48fb-921a-06d332f8f5fb" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"StartTime\": \"2026-06-10T17:12:10.3374556Z\",\r\n \"EndTime\": \"2026-06-10T17:12:13.8319517Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentNullException: Value cannot be null. (Parameter 'DriveItem')\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:14.0905943Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-a1c2-a0228f5e60cb\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T17:12:10.4261792Z\",\r\n \"EndTime\": \"2026-06-10T17:12:10.4653291Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:11.0505953Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9ff8-913f61a075e5\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"Name\": \"Find Files And Folders\",\r\n \"StartTime\": \"2026-06-10T17:12:10.6087938Z\",\r\n \"EndTime\": \"2026-06-10T17:12:10.6096896Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.FindFilesAndFolders\\\",\\\"Private.Query\\\":\\\"no-such-file-zzz-repro\\\",\\\"Private.First\\\":\\\"foundItem\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Find Files And Folders\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:11.0505953Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-a1c9-53ef7ebd7f3a\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"Name\": \"Copy Item\",\r\n \"StartTime\": \"2026-06-10T17:12:13.7782756Z\",\r\n \"EndTime\": \"2026-06-10T17:12:13.7790144Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.CopyItem\\\",\\\"Private.DriveItem\\\":\\\"foundItem\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Copy Item\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:14.0905943Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 24,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "909ed39c-08a1-48fb-921a-06d332f8f5fb" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"14bcebd6-6eeb-4933-ade8-947658d27b70\",\r\n \"Info\": \"Value cannot be null. (Parameter 'DriveItem')\\r\\n\\r\\nin Main.xaml\\r\\n at CopyItem \\\"Copy Item\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Copy Item null DriveItem repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'DriveItem')\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\",\r\n \"ReleaseName\": \"DriveItemCopier\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53467,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"909ed39c08a148fb921a06d332f8f5fb\",\r\n \"Id\": 4015106\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "909ed39c-08a1-48fb-921a-06d332f8f5fb" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T17:12:13.804Z\",\r\n \"Message\": \"Copy Item: Value cannot be null. (Parameter 'DriveItem')\\r\\nin Main.xaml\\r\\n at CopyItem \\\"Copy Item\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Copy Item null DriveItem repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "909ed39c-08a1-48fb-921a-06d332f8f5fb" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T17:12:14.253Z\",\r\n \"JobId\": 4015106\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T17:12:06.217Z\",\r\n \"JobId\": 4015106\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T16:53:17.913Z\",\r\n \"JobId\": 4015106\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "909ed39c-08a1-48fb-921a-06d332f8f5fb" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"StartTime\": \"2026-06-10T17:12:10.3374556Z\",\r\n \"EndTime\": \"2026-06-10T17:12:13.8319517Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentNullException: Value cannot be null. (Parameter 'DriveItem')\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Files.CopyItem.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:14.0905943Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-a1c2-a0228f5e60cb\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T17:12:10.4261792Z\",\r\n \"EndTime\": \"2026-06-10T17:12:10.4653291Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:11.0505953Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9ff8-913f61a075e5\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"Name\": \"Find Files And Folders\",\r\n \"StartTime\": \"2026-06-10T17:12:10.6087938Z\",\r\n \"EndTime\": \"2026-06-10T17:12:10.6096896Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.FindFilesAndFolders\\\",\\\"Private.Query\\\":\\\"no-such-file-zzz-repro\\\",\\\"Private.First\\\":\\\"foundItem\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Find Files And Folders\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:11.0505953Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-a1c9-53ef7ebd7f3a\",\r\n \"TraceId\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ParentId\": \"00000000-0000-0000-8776-0214877dc4e4\",\r\n \"Name\": \"Copy Item\",\r\n \"StartTime\": \"2026-06-10T17:12:13.7782756Z\",\r\n \"EndTime\": \"2026-06-10T17:12:13.7790144Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.CopyItem\\\",\\\"Private.DriveItem\\\":\\\"foundItem\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Copy Item\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ec2c2000-846e-4020-ba3f-297f3dfce24e\",\r\n \"JobKey\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:12:14.0905943Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 24,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 909ed39c-08a1-48fb-921a-06d332f8f5fb", + "min": 1, + "description": "Agent must fetch the failing job to read the raw ArgumentNullException (Parameter 'DriveItem')." + }, + { + "pattern": "traces", + "min": 1, + "description": "Trace spans carry the variable-binding chain (FindFilesAndFolders First='foundItem' -> CopyItem DriveItem='foundItem'); the agent may retrieve them via `or jobs traces ` or `traces spans get --job-key `." + }, + { + "pattern": "or jobs logs 909ed39c-08a1-48fb-921a-06d332f8f5fb", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history 909ed39c-08a1-48fb-921a-06d332f8f5fb", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/task.yaml index 3053f73f73..beedaaf0fe 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-copyitem-null-driveitem/task.yaml @@ -19,21 +19,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/README.md new file mode 100644 index 0000000000..34edd40219 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/README.md @@ -0,0 +1,58 @@ +# O365 Create Folder Invalid Path — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `DriveFolderProvisioner` (key +`a04406a9-4c64-40d0-a7f3-042c3224c13b`), faulted with +`UiPath.MicrosoftOffice365.Office365Exception: Folder path segment +' Quarterly' cannot have leading or trailing whitespace. (Parameter +'FolderPath')` thrown by the Create Folder (`CreateFolderConnections`) +activity in `Main.xaml`. The package's own pre-flight +path validation in `GraphServiceClientProxy.CreateFolderByPathAsync` threw +(exception chain `Office365Exception → Office365InternalException → +ArgumentException`) before any Microsoft Graph folder operation ran — a +deterministic configuration error, not auth, not a name conflict, not a +parent-not-found. Fix: trim the whitespace-padded segment in the configured +`FolderPath` (e.g. `Reports/Quarterly`). + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info/logs carry the full fault stack). + +No trace spans were recorded for this job, so both trace forms (`or jobs +traces ` and `traces spans get --job-key `) are mocked with a +well-formed empty `JobTraces` response so the agent reads "no spans" instead +of misreading a bare `[]`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs (the generator's transcript pass is unreliable for this session). If +the source investigation changes, update the fixture JSONs directly and re-run +the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-createfolder-invalid-path --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/1862b496ba.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/1862b496ba.json deleted file mode 100644 index 44bfb8e392..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/1862b496ba.json +++ /dev/null @@ -1,404 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 28, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/26dc3a8bfc.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/26dc3a8bfc.json deleted file mode 100644 index d6e7824a39..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/26dc3a8bfc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T18:59:39.693Z", - "JobId": 4015112 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T18:59:36.943Z", - "JobId": 4015112 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T18:59:36.400Z", - "JobId": 4015112 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/5ba7fc79b9.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/5ba7fc79b9.json deleted file mode 100644 index 05866444bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/5ba7fc79b9.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "080a2e57-d62d-4a52-a9cc-eb9e5ec8e305", - "Info": "Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\r\n\r\nin Main.xaml\r\n at CreateFolderConnections \"Create Folder\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Create Folder invalid path repro\"\r\n at Main \"Main\"\r\n\r\nUiPath.MicrosoftOffice365.Office365Exception: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\r\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\r\n ---> System.ArgumentException: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderByPathAsync(GraphServiceClient client, CreateFolderRequest request, DriveItem parentDriveItem, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderAsync(CreateFolderRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Files.CreateFolderConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderByPathAsync(GraphServiceClient client, CreateFolderRequest request, DriveItem parentDriveItem, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderAsync(CreateFolderRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Files.CreateFolderConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T18:59:36.350Z", - "ReleaseName": "DriveFolderProvisioner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53471, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T18:59:39.683Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a04406a94c6440d0a7f3042c3224c13b", - "Id": 4015112 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/969f48072f.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/969f48072f.json deleted file mode 100644 index e48011cba8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/969f48072f.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T18:59:39.115Z", - "Message": "Create Folder: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\r\nin Main.xaml\r\n at CreateFolderConnections \"Create Folder\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Create Folder invalid path repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.MicrosoftOffice365.Office365Exception\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/README.md deleted file mode 100644 index a66a097a6c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# O365 Create Folder Invalid Path — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `DriveFolderProvisioner` (key -`a04406a9-4c64-40d0-a7f3-042c3224c13b`), faulted with -`UiPath.MicrosoftOffice365.Office365Exception: Folder path segment -' Quarterly' cannot have leading or trailing whitespace. (Parameter -'FolderPath')` thrown by the Create Folder (`CreateFolderConnections`) -activity in `Main.xaml`. The package's own pre-flight -path validation in `GraphServiceClientProxy.CreateFolderByPathAsync` threw -(exception chain `Office365Exception → Office365InternalException → -ArgumentException`) before any Microsoft Graph folder operation ran — a -deterministic configuration error, not auth, not a name conflict, not a -parent-not-found. Fix: trim the whitespace-padded segment in the configured -`FolderPath` (e.g. `Reports/Quarterly`). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info/logs carry the full fault stack). - -No trace spans were recorded for this job, so both trace forms (`or jobs -traces ` and `traces spans get --job-key `) are mocked with a -well-formed empty `JobTraces` response so the agent reads "no spans" instead -of misreading a bare `[]`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs (the generator's transcript pass is unreliable for this session). If -the source investigation changes, update the fixture JSONs directly and re-run -the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-createfolder-invalid-path --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/manifest.json deleted file mode 100644 index 636a5f1ef3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/m/r/manifest.json +++ /dev/null @@ -1,87 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get a04406a9-4c64-40d0-a7f3-042c3224c13b", - "file": "5ba7fc79b9.json" - }, - { - "match": "or jobs logs a04406a9-4c64-40d0-a7f3-042c3224c13b --level Error", - "file": "969f48072f.json" - }, - { - "match": "or jobs logs a04406a9-4c64-40d0-a7f3-042c3224c13b", - "file": "969f48072f.json" - }, - { - "match": "or jobs history a04406a9-4c64-40d0-a7f3-042c3224c13b", - "file": "26dc3a8bfc.json" - }, - { - "match": "or jobs traces a04406a9-4c64-40d0-a7f3-042c3224c13b", - "file": "2d930e2440.json", - "_doc": "No trace spans were recorded for this job — the fixture is a well-formed empty JobTraces response so the agent reads 'no spans' instead of misreading a bare []. The root cause is fully visible in jobs get / jobs logs." - }, - { - "match": "traces spans get --job-key a04406a9-4c64-40d0-a7f3-042c3224c13b", - "file": "2d930e2440.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "1862b496ba.json" - }, - { - "match": "or jobs list", - "file": "1862b496ba.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get a04406a9-4c64-40d0-a7f3-042c3224c13b", - "min": 1, - "description": "Agent must fetch the failing job to read the Office365Exception: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath') from CreateFolderConnections, with the GraphServiceClientProxy.CreateFolderByPathAsync pre-flight validation stack." - }, - { - "pattern": "or jobs logs a04406a9-4c64-40d0-a7f3-042c3224c13b", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "traces", - "min": 0, - "description": "OPTIONAL — no spans were recorded for this job; the empty response adds nothing. Root cause is already fully visible in jobs get." - }, - { - "pattern": "or jobs history a04406a9-4c64-40d0-a7f3-042c3224c13b", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/uip-fixture.json new file mode 100644 index 0000000000..33719c644e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/data/uip-fixture.json @@ -0,0 +1,142 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "a04406a9-4c64-40d0-a7f3-042c3224c13b", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T18:59:39.115Z\",\r\n \"Message\": \"Create Folder: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\\r\\nin Main.xaml\\r\\n at CreateFolderConnections \\\"Create Folder\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Create Folder invalid path repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.MicrosoftOffice365.Office365Exception\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "a04406a9-4c64-40d0-a7f3-042c3224c13b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 28,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a04406a9-4c64-40d0-a7f3-042c3224c13b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"080a2e57-d62d-4a52-a9cc-eb9e5ec8e305\",\r\n \"Info\": \"Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\\r\\n\\r\\nin Main.xaml\\r\\n at CreateFolderConnections \\\"Create Folder\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Create Folder invalid path repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.MicrosoftOffice365.Office365Exception: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\\r\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\\r\\n ---> System.ArgumentException: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderByPathAsync(GraphServiceClient client, CreateFolderRequest request, DriveItem parentDriveItem, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderAsync(CreateFolderRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.CreateFolderConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderByPathAsync(GraphServiceClient client, CreateFolderRequest request, DriveItem parentDriveItem, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.CreateFolderAsync(CreateFolderRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.CreateFolderConnections.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\",\r\n \"ReleaseName\": \"DriveFolderProvisioner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53471,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a04406a94c6440d0a7f3042c3224c13b\",\r\n \"Id\": 4015112\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a04406a9-4c64-40d0-a7f3-042c3224c13b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T18:59:39.115Z\",\r\n \"Message\": \"Create Folder: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath')\\r\\nin Main.xaml\\r\\n at CreateFolderConnections \\\"Create Folder\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Create Folder invalid path repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.MicrosoftOffice365.Office365Exception\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "a04406a9-4c64-40d0-a7f3-042c3224c13b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T18:59:39.693Z\",\r\n \"JobId\": 4015112\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T18:59:36.943Z\",\r\n \"JobId\": 4015112\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T18:59:36.400Z\",\r\n \"JobId\": 4015112\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "a04406a9-4c64-40d0-a7f3-042c3224c13b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 28,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get a04406a9-4c64-40d0-a7f3-042c3224c13b", + "min": 1, + "description": "Agent must fetch the failing job to read the Office365Exception: Folder path segment ' Quarterly' cannot have leading or trailing whitespace. (Parameter 'FolderPath') from CreateFolderConnections, with the GraphServiceClientProxy.CreateFolderByPathAsync pre-flight validation stack." + }, + { + "pattern": "or jobs logs a04406a9-4c64-40d0-a7f3-042c3224c13b", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "traces", + "min": 0, + "description": "OPTIONAL — no spans were recorded for this job; the empty response adds nothing. Root cause is already fully visible in jobs get." + }, + { + "pattern": "or jobs history a04406a9-4c64-40d0-a7f3-042c3224c13b", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/task.yaml index ca5ad92943..1aa460967f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-createfolder-invalid-path/task.yaml @@ -21,21 +21,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/README.md new file mode 100644 index 0000000000..53fb3bcabd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/README.md @@ -0,0 +1,50 @@ +# O365 Download File Folder Item — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `ReportDownloader` (key `64580780-4aa1-4ec2-b5f2-abd5439ea200`), +faulted with `UiPath.MicrosoftOffice365.Office365Exception: Folders cannot be +downloaded with this activity. Please input a different DriveItem.` thrown by +the legacy O365 `DownloadFile` activity. Job trace spans proved the binding +chain: an upstream `FindFilesAndFolders` ("Find a folder by name", query +`Documents`, `First` output bound to `foundFolder`) found the OneDrive folder +'Documents'; `DownloadFile` was invoked with `File = foundFolder`, +`LocalFilePath = C:\Temp` — a folder bound where a file is required. The +scope and the find activity completed normally (auth and permissions fine). +Fix: point the activity at a file DriveItem, and/or iterate the folder's +contents (For Each File/Folder) downloading per file; a type guard between +the search and the download prevents recurrence. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job traces carried the variable-binding chain). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +If the source investigation changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-download-folder-item --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/2cd3388d10.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/2cd3388d10.json deleted file mode 100644 index 1f7de31b60..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/2cd3388d10.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "6cf2ce57-bafe-4b01-aa14-864c31136a02", - "Info": "Folders cannot be downloaded with this activity. Please input a different DriveItem.\r\n\r\nin Main.xaml\r\n at DownloadFile \"Download File\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Download conversion-or-destination repro\"\r\n at Main \"Main\"\r\n\r\nUiPath.MicrosoftOffice365.Office365Exception: Folders cannot be downloaded with this activity. Please input a different DriveItem.\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T19:09:03.457Z", - "ReleaseName": "ReportDownloader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53472, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T19:09:09.557Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "645807804aa14ec2b5f2abd5439ea200", - "Id": 4015113 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/88a9723f86.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/88a9723f86.json deleted file mode 100644 index d9f30b11d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/88a9723f86.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T19:09:09.567Z", - "JobId": 4015113 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T19:09:03.777Z", - "JobId": 4015113 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T19:09:03.483Z", - "JobId": 4015113 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/8b01b1cd5f.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/8b01b1cd5f.json deleted file mode 100644 index c17261203f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/8b01b1cd5f.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T19:09:09.062Z", - "Message": "Download File: Folders cannot be downloaded with this activity. Please input a different DriveItem.\r\nin Main.xaml\r\n at DownloadFile \"Download File\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Download conversion-or-destination repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.MicrosoftOffice365.Office365Exception\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/README.md deleted file mode 100644 index ce0810d350..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# O365 Download File Folder Item — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `ReportDownloader` (key `64580780-4aa1-4ec2-b5f2-abd5439ea200`), -faulted with `UiPath.MicrosoftOffice365.Office365Exception: Folders cannot be -downloaded with this activity. Please input a different DriveItem.` thrown by -the legacy O365 `DownloadFile` activity. Job trace spans proved the binding -chain: an upstream `FindFilesAndFolders` ("Find a folder by name", query -`Documents`, `First` output bound to `foundFolder`) found the OneDrive folder -'Documents'; `DownloadFile` was invoked with `File = foundFolder`, -`LocalFilePath = C:\Temp` — a folder bound where a file is required. The -scope and the find activity completed normally (auth and permissions fine). -Fix: point the activity at a file DriveItem, and/or iterate the folder's -contents (For Each File/Folder) downloading per file; a type guard between -the search and the download prevents recurrence. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job traces carried the variable-binding chain). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -If the source investigation changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-download-folder-item --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/bdc90e50e5.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/bdc90e50e5.json deleted file mode 100644 index 27ffb188d8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/bdc90e50e5.json +++ /dev/null @@ -1,418 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 29, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/cba960786e.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/cba960786e.json deleted file mode 100644 index b83166f18a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/cba960786e.json +++ /dev/null @@ -1,110 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-59c4-45d971ff518b", - "TraceId": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ParentId": null, - "Name": "Robot Job 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "StartTime": "2026-06-10T19:09:04.2077017Z", - "EndTime": "2026-06-10T19:09:09.1094331Z", - "Attributes": "{\"Exception\":\"UiPath.MicrosoftOffice365.Office365Exception: Folders cannot be downloaded with this activity. Please input a different DriveItem.\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "e9e772df-55a4-4b46-9d0b-1965d2f8a9f5", - "JobKey": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:09:09.4161752Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-9e6a-831815053f3a", - "TraceId": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ParentId": "00000000-0000-0000-59c4-45d971ff518b", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-10T19:09:04.3093204Z", - "EndTime": "2026-06-10T19:09:04.3478129Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"PropertiesPanel\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "e9e772df-55a4-4b46-9d0b-1965d2f8a9f5", - "JobKey": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:09:05.1862226Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-d215-08c655977ef5", - "TraceId": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ParentId": "00000000-0000-0000-59c4-45d971ff518b", - "Name": "Find a folder by name (point Query at an existing OneDrive folder)", - "StartTime": "2026-06-10T19:09:04.6178641Z", - "EndTime": "2026-06-10T19:09:04.6187416Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Files.FindFilesAndFolders\",\"Private.Query\":\"Documents\",\"Private.First\":\"foundFolder\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Find a folder by name (point Query at an existing OneDrive folder)\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "e9e772df-55a4-4b46-9d0b-1965d2f8a9f5", - "JobKey": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:09:05.1862226Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-7558-472b41db178d", - "TraceId": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ParentId": "00000000-0000-0000-59c4-45d971ff518b", - "Name": "Download File", - "StartTime": "2026-06-10T19:09:09.0163014Z", - "EndTime": "2026-06-10T19:09:09.0167582Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Files.DownloadFile\",\"Private.LocalFilePath\":\"C:\\\\Temp\",\"Private.File\":\"foundFolder\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Download File\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "e9e772df-55a4-4b46-9d0b-1965d2f8a9f5", - "JobKey": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:09:09.4161752Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/manifest.json deleted file mode 100644 index c592e98256..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/m/r/manifest.json +++ /dev/null @@ -1,86 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "file": "2cd3388d10.json" - }, - { - "match": "or jobs logs 64580780-4aa1-4ec2-b5f2-abd5439ea200 --level Error", - "file": "8b01b1cd5f.json" - }, - { - "match": "or jobs logs 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "file": "8b01b1cd5f.json" - }, - { - "match": "or jobs history 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "file": "88a9723f86.json" - }, - { - "match": "or jobs traces 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "file": "cba960786e.json" - }, - { - "match": "traces spans get --job-key 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "file": "cba960786e.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same span payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default and the agent loses the FindFilesAndFolders -> DownloadFile binding-chain evidence." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "bdc90e50e5.json" - }, - { - "match": "or jobs list", - "file": "bdc90e50e5.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "min": 1, - "description": "Agent must fetch the failing job to read the Office365Exception (Folders cannot be downloaded with this activity. Please input a different DriveItem.) from DownloadFile.ExecuteAsync." - }, - { - "pattern": "traces", - "min": 1, - "description": "Trace spans carry the variable-binding chain (FindFilesAndFolders Query='Documents' First='foundFolder' -> DownloadFile File='foundFolder' LocalFilePath='C:\\Temp'); the agent may retrieve them via `or jobs traces ` or `traces spans get --job-key `." - }, - { - "pattern": "or jobs logs 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history 64580780-4aa1-4ec2-b5f2-abd5439ea200", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/uip-fixture.json new file mode 100644 index 0000000000..fab5d01590 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/data/uip-fixture.json @@ -0,0 +1,142 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "64580780-4aa1-4ec2-b5f2-abd5439ea200", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T19:09:09.062Z\",\r\n \"Message\": \"Download File: Folders cannot be downloaded with this activity. Please input a different DriveItem.\\r\\nin Main.xaml\\r\\n at DownloadFile \\\"Download File\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Download conversion-or-destination repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.MicrosoftOffice365.Office365Exception\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "64580780-4aa1-4ec2-b5f2-abd5439ea200" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"StartTime\": \"2026-06-10T19:09:04.2077017Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.1094331Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.MicrosoftOffice365.Office365Exception: Folders cannot be downloaded with this activity. Please input a different DriveItem.\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:09.4161752Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9e6a-831815053f3a\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T19:09:04.3093204Z\",\r\n \"EndTime\": \"2026-06-10T19:09:04.3478129Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:05.1862226Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-d215-08c655977ef5\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"Name\": \"Find a folder by name (point Query at an existing OneDrive folder)\",\r\n \"StartTime\": \"2026-06-10T19:09:04.6178641Z\",\r\n \"EndTime\": \"2026-06-10T19:09:04.6187416Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.FindFilesAndFolders\\\",\\\"Private.Query\\\":\\\"Documents\\\",\\\"Private.First\\\":\\\"foundFolder\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Find a folder by name (point Query at an existing OneDrive folder)\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:05.1862226Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-7558-472b41db178d\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"Name\": \"Download File\",\r\n \"StartTime\": \"2026-06-10T19:09:09.0163014Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.0167582Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.DownloadFile\\\",\\\"Private.LocalFilePath\\\":\\\"C:\\\\\\\\Temp\\\",\\\"Private.File\\\":\\\"foundFolder\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Download File\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:09.4161752Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 29,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "64580780-4aa1-4ec2-b5f2-abd5439ea200" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"6cf2ce57-bafe-4b01-aa14-864c31136a02\",\r\n \"Info\": \"Folders cannot be downloaded with this activity. Please input a different DriveItem.\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFile \\\"Download File\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Download conversion-or-destination repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.MicrosoftOffice365.Office365Exception: Folders cannot be downloaded with this activity. Please input a different DriveItem.\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\",\r\n \"ReleaseName\": \"ReportDownloader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53472,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"645807804aa14ec2b5f2abd5439ea200\",\r\n \"Id\": 4015113\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "64580780-4aa1-4ec2-b5f2-abd5439ea200" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T19:09:09.062Z\",\r\n \"Message\": \"Download File: Folders cannot be downloaded with this activity. Please input a different DriveItem.\\r\\nin Main.xaml\\r\\n at DownloadFile \\\"Download File\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Download conversion-or-destination repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.MicrosoftOffice365.Office365Exception\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "64580780-4aa1-4ec2-b5f2-abd5439ea200" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T19:09:09.567Z\",\r\n \"JobId\": 4015113\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T19:09:03.777Z\",\r\n \"JobId\": 4015113\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T19:09:03.483Z\",\r\n \"JobId\": 4015113\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "64580780-4aa1-4ec2-b5f2-abd5439ea200" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"StartTime\": \"2026-06-10T19:09:04.2077017Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.1094331Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.MicrosoftOffice365.Office365Exception: Folders cannot be downloaded with this activity. Please input a different DriveItem.\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.DownloadFileAsync(DownloadFileRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Files.DownloadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:09.4161752Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9e6a-831815053f3a\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T19:09:04.3093204Z\",\r\n \"EndTime\": \"2026-06-10T19:09:04.3478129Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:05.1862226Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-d215-08c655977ef5\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"Name\": \"Find a folder by name (point Query at an existing OneDrive folder)\",\r\n \"StartTime\": \"2026-06-10T19:09:04.6178641Z\",\r\n \"EndTime\": \"2026-06-10T19:09:04.6187416Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.FindFilesAndFolders\\\",\\\"Private.Query\\\":\\\"Documents\\\",\\\"Private.First\\\":\\\"foundFolder\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Find a folder by name (point Query at an existing OneDrive folder)\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:05.1862226Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-7558-472b41db178d\",\r\n \"TraceId\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ParentId\": \"00000000-0000-0000-59c4-45d971ff518b\",\r\n \"Name\": \"Download File\",\r\n \"StartTime\": \"2026-06-10T19:09:09.0163014Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.0167582Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Files.DownloadFile\\\",\\\"Private.LocalFilePath\\\":\\\"C:\\\\\\\\Temp\\\",\\\"Private.File\\\":\\\"foundFolder\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Download File\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e9e772df-55a4-4b46-9d0b-1965d2f8a9f5\",\r\n \"JobKey\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:09:09.4161752Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 29,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 64580780-4aa1-4ec2-b5f2-abd5439ea200", + "min": 1, + "description": "Agent must fetch the failing job to read the Office365Exception (Folders cannot be downloaded with this activity. Please input a different DriveItem.) from DownloadFile.ExecuteAsync." + }, + { + "pattern": "traces", + "min": 1, + "description": "Trace spans carry the variable-binding chain (FindFilesAndFolders Query='Documents' First='foundFolder' -> DownloadFile File='foundFolder' LocalFilePath='C:\\Temp'); the agent may retrieve them via `or jobs traces ` or `traces spans get --job-key `." + }, + { + "pattern": "or jobs logs 64580780-4aa1-4ec2-b5f2-abd5439ea200", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history 64580780-4aa1-4ec2-b5f2-abd5439ea200", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/task.yaml index efd8262f3c..3c6e374fe4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-download-folder-item/task.yaml @@ -21,21 +21,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/README.md new file mode 100644 index 0000000000..4860becf5a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/README.md @@ -0,0 +1,51 @@ +# O365 Forward Mail Null Message — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `MailForwarder` (key `50bcdb33-f7a8-42d4-a52d-ebbeeb135e99`), +faulted with a raw `System.NullReferenceException: Object reference not set +to an instance of an object.` thrown by the legacy O365 `ForwardMail` +activity (`ForwardMail.ExecuteAsync` top frame). Job trace spans proved the +cause: the activity's `Message` input is bound to variable `originalEmail`, +but only 3 spans exist (job root → Microsoft 365 Scope → Forward Mail) — no +upstream mail-fetch/producer activity ever ran, so `originalEmail` stayed +null. The `To` attribute is a non-null literal, ruling out the +null-recipient-element cause; the fault landed ~1.3 ms into the activity, +before any Microsoft Graph call. Fix: assign/produce the message before +forwarding and/or add a null guard on `originalEmail`; migrating to the +Connections Forward Email is a complementary preventive fix. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job traces carried the binding + missing-producer +evidence). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +If the source investigation changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-forwardmail-null-message --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/4e870d6910.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/4e870d6910.json deleted file mode 100644 index eedad90eef..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/4e870d6910.json +++ /dev/null @@ -1,236 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 16, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/4f0020f3a3.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/4f0020f3a3.json deleted file mode 100644 index d72044dcd7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/4f0020f3a3.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-52d5-f09e8cb3a875", - "TraceId": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "ParentId": null, - "Name": "Robot Job 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "StartTime": "2026-06-10T17:58:42.1352773Z", - "EndTime": "2026-06-10T17:58:42.9509869Z", - "Attributes": "{\"Exception\":\"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "681f9a0e-c823-45b4-b15e-c51d6a1c2c77", - "JobKey": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:58:43.2083124Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-2bf1-23e7945c3b89", - "TraceId": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "ParentId": "00000000-0000-0000-52d5-f09e8cb3a875", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-10T17:58:42.2599288Z", - "EndTime": "2026-06-10T17:58:42.3005767Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"PropertiesPanel\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "681f9a0e-c823-45b4-b15e-c51d6a1c2c77", - "JobKey": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:58:42.944309Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-9075-cc809dd36834", - "TraceId": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "ParentId": "00000000-0000-0000-52d5-f09e8cb3a875", - "Name": "Forward Mail", - "StartTime": "2026-06-10T17:58:42.4524518Z", - "EndTime": "2026-06-10T17:58:42.4537581Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail\",\"Public.IsDraft\":\"False\",\"Private.Message\":\"originalEmail\",\"Private.NewSubject\":\"FW: NRE repro\",\"Public.ReplaceExisting\":\"False\",\"Private.To\":\"new string[] { \\\"recipient@example.com\\\" }\",\"Public.IsBodyHTML\":\"False\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Forward Mail\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "681f9a0e-c823-45b4-b15e-c51d6a1c2c77", - "JobKey": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T17:58:42.944309Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/README.md deleted file mode 100644 index 8fe051d956..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/README.md +++ /dev/null @@ -1,50 +0,0 @@ -# O365 Forward Mail Null Message — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `MailForwarder` (key `50bcdb33-f7a8-42d4-a52d-ebbeeb135e99`), -faulted with a raw `System.NullReferenceException: Object reference not set -to an instance of an object.` thrown by the legacy O365 `ForwardMail` -activity (`ForwardMail.ExecuteAsync` top frame). Job trace spans proved the -cause: the activity's `Message` input is bound to variable `originalEmail`, -but only 3 spans exist (job root → Microsoft 365 Scope → Forward Mail) — no -upstream mail-fetch/producer activity ever ran, so `originalEmail` stayed -null. The `To` attribute is a non-null literal, ruling out the -null-recipient-element cause; the fault landed ~1.3 ms into the activity, -before any Microsoft Graph call. Fix: assign/produce the message before -forwarding and/or add a null guard on `originalEmail`; migrating to the -Connections Forward Email is a complementary preventive fix. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job traces carried the binding + missing-producer -evidence). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -If the source investigation changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-forwardmail-null-message --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/a494b1656a.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/a494b1656a.json deleted file mode 100644 index de36d728c2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/a494b1656a.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "78583fa2-6eb0-4a28-8b83-a62da737c620", - "Info": "Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at ForwardMail \"Forward Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 legacy Mail NullReferenceException repro\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T17:58:41.137Z", - "ReleaseName": "MailForwarder", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53468, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T17:58:43.360Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "50bcdb33f7a842d4a52debbeeb135e99", - "Id": 4015109 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/aa675830de.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/aa675830de.json deleted file mode 100644 index e6fe36a685..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/aa675830de.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T17:58:43.370Z", - "JobId": 4015109 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T17:58:41.690Z", - "JobId": 4015109 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T17:58:41.193Z", - "JobId": 4015109 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/e4e123f4fb.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/e4e123f4fb.json deleted file mode 100644 index 282a3bbd88..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/e4e123f4fb.json +++ /dev/null @@ -1,362 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 25, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/ee002bb2c4.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/ee002bb2c4.json deleted file mode 100644 index 0ffc2a7dfd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/ee002bb2c4.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-10T17:58:43.265Z", - "Message": "MailForwarder execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T17:58:42.919Z", - "Message": "Forward Mail: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at ForwardMail \"Forward Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 legacy Mail NullReferenceException repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.NullReferenceException\r\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-10T17:58:42.151Z", - "Message": "MailForwarder execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/manifest.json deleted file mode 100644 index dc0b3c6273..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/m/r/manifest.json +++ /dev/null @@ -1,88 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "file": "a494b1656a.json" - }, - { - "match": "or jobs logs 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "file": "ee002bb2c4.json", - "_doc": "Full log set (Info + Error). Also serves --level Error queries — the Error entry carries the full exception stack either way." - }, - { - "match": "or jobs history 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "file": "aa675830de.json" - }, - { - "match": "or jobs traces 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "file": "4f0020f3a3.json" - }, - { - "match": "traces spans get --job-key 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "file": "4f0020f3a3.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same span payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default and the agent concludes no upstream activity existed for the wrong reason (degraded replicate observed on a sibling scenario)." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "4e870d6910.json", - "_doc": "Faulted-only job list recorded from the original session. Placed before the generic jobs-list rules; other flag orderings fall through to the full job list, whose newest entry is the same faulted job." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "e4e123f4fb.json" - }, - { - "match": "or jobs list", - "file": "e4e123f4fb.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "min": 1, - "description": "Agent must fetch the failing job to read the raw NullReferenceException from ForwardMail.ExecuteAsync." - }, - { - "pattern": "traces", - "min": 1, - "description": "Trace spans carry the binding evidence (ForwardMail Message='originalEmail', To=non-null literal, only 3 spans / no producer activity); the agent may retrieve them via `or jobs traces ` or `traces spans get --job-key `." - }, - { - "pattern": "or jobs logs 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/uip-fixture.json new file mode 100644 index 0000000000..d3b1ea8fdc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/data/uip-fixture.json @@ -0,0 +1,143 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 16,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-52d5-f09e8cb3a875\",\r\n \"TraceId\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"StartTime\": \"2026-06-10T17:58:42.1352773Z\",\r\n \"EndTime\": \"2026-06-10T17:58:42.9509869Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.NullReferenceException: Object reference not set to an instance of an object.\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"681f9a0e-c823-45b4-b15e-c51d6a1c2c77\",\r\n \"JobKey\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:58:43.2083124Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-2bf1-23e7945c3b89\",\r\n \"TraceId\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ParentId\": \"00000000-0000-0000-52d5-f09e8cb3a875\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T17:58:42.2599288Z\",\r\n \"EndTime\": \"2026-06-10T17:58:42.3005767Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"681f9a0e-c823-45b4-b15e-c51d6a1c2c77\",\r\n \"JobKey\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:58:42.944309Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9075-cc809dd36834\",\r\n \"TraceId\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ParentId\": \"00000000-0000-0000-52d5-f09e8cb3a875\",\r\n \"Name\": \"Forward Mail\",\r\n \"StartTime\": \"2026-06-10T17:58:42.4524518Z\",\r\n \"EndTime\": \"2026-06-10T17:58:42.4537581Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail\\\",\\\"Public.IsDraft\\\":\\\"False\\\",\\\"Private.Message\\\":\\\"originalEmail\\\",\\\"Private.NewSubject\\\":\\\"FW: NRE repro\\\",\\\"Public.ReplaceExisting\\\":\\\"False\\\",\\\"Private.To\\\":\\\"new string[] { \\\\\\\"recipient@example.com\\\\\\\" }\\\",\\\"Public.IsBodyHTML\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Forward Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"681f9a0e-c823-45b4-b15e-c51d6a1c2c77\",\r\n \"JobKey\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:58:42.944309Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 25,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"78583fa2-6eb0-4a28-8b83-a62da737c620\",\r\n \"Info\": \"Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at ForwardMail \\\"Forward Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 legacy Mail NullReferenceException repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\",\r\n \"ReleaseName\": \"MailForwarder\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53468,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"50bcdb33f7a842d4a52debbeeb135e99\",\r\n \"Id\": 4015109\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T17:58:43.265Z\",\r\n \"Message\": \"MailForwarder execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T17:58:42.919Z\",\r\n \"Message\": \"Forward Mail: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at ForwardMail \\\"Forward Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 legacy Mail NullReferenceException repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T17:58:42.151Z\",\r\n \"Message\": \"MailForwarder execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T17:58:43.370Z\",\r\n \"JobId\": 4015109\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T17:58:41.690Z\",\r\n \"JobId\": 4015109\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T17:58:41.193Z\",\r\n \"JobId\": 4015109\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-52d5-f09e8cb3a875\",\r\n \"TraceId\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"StartTime\": \"2026-06-10T17:58:42.1352773Z\",\r\n \"EndTime\": \"2026-06-10T17:58:42.9509869Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.NullReferenceException: Object reference not set to an instance of an object.\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"681f9a0e-c823-45b4-b15e-c51d6a1c2c77\",\r\n \"JobKey\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:58:43.2083124Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-2bf1-23e7945c3b89\",\r\n \"TraceId\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ParentId\": \"00000000-0000-0000-52d5-f09e8cb3a875\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T17:58:42.2599288Z\",\r\n \"EndTime\": \"2026-06-10T17:58:42.3005767Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"681f9a0e-c823-45b4-b15e-c51d6a1c2c77\",\r\n \"JobKey\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:58:42.944309Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9075-cc809dd36834\",\r\n \"TraceId\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ParentId\": \"00000000-0000-0000-52d5-f09e8cb3a875\",\r\n \"Name\": \"Forward Mail\",\r\n \"StartTime\": \"2026-06-10T17:58:42.4524518Z\",\r\n \"EndTime\": \"2026-06-10T17:58:42.4537581Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.ForwardMail\\\",\\\"Public.IsDraft\\\":\\\"False\\\",\\\"Private.Message\\\":\\\"originalEmail\\\",\\\"Private.NewSubject\\\":\\\"FW: NRE repro\\\",\\\"Public.ReplaceExisting\\\":\\\"False\\\",\\\"Private.To\\\":\\\"new string[] { \\\\\\\"recipient@example.com\\\\\\\" }\\\",\\\"Public.IsBodyHTML\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Forward Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"681f9a0e-c823-45b4-b15e-c51d6a1c2c77\",\r\n \"JobKey\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T17:58:42.944309Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 25,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", + "min": 1, + "description": "Agent must fetch the failing job to read the raw NullReferenceException from ForwardMail.ExecuteAsync." + }, + { + "pattern": "traces", + "min": 1, + "description": "Trace spans carry the binding evidence (ForwardMail Message='originalEmail', To=non-null literal, only 3 spans / no producer activity); the agent may retrieve them via `or jobs traces ` or `traces spans get --job-key `." + }, + { + "pattern": "or jobs logs 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history 50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/task.yaml index ecf64bea72..842c5c8fb1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-forwardmail-null-message/task.yaml @@ -19,21 +19,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/README.md new file mode 100644 index 0000000000..b7a042cd6b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/README.md @@ -0,0 +1,103 @@ +# O365 Get Mail — Mail Folder Not Found (unwrapped ArgumentNullException) — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `TeamMailRetriever` (key `e893a4d4-e061-4cba-a799-6ed4de6ee4e2`), +faulted at the legacy **Get Mail** activity ("Get Mail from nonexistent +folder" in `Main.xaml`) inside the Microsoft 365 Scope +with the unwrapped `System.ArgumentNullException: Value cannot be null. +(Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on +this account.')` — the unwrapped form of the mail-folder-not-found +signature; the inner sentence is the match. Faulting frame: +`GraphServiceClientExtensions.GetMailFolder(...)` during folder resolution, +before any message retrieval. The activity's `MailFolder` argument was the +literal name `NoSuchFolder-Repro-123`; folder enumeration of the mailbox +resolved by the scope (interactive-token auth, no shared-mailbox/account +override — the authenticated user's own mailbox) found no match. Fix: +verify/correct the folder name or the `MailFolder` argument (path form for +nested folders), confirm the resolved mailbox, then restart the job. + +### Span evidence pattern + +This scenario's cause disambiguation lives in the **trace spans**: the +"Get Mail" `GetMail` span carries +`Private.MailFolder="NoSuchFolder-Repro-123"` (matching the error verbatim) +and `Public.AuthScopesInvalid="False"` (eliminating the insufficient-scope +branch), and the Microsoft 365 Scope span shows +`AuthenticationType=InteractiveToken` with no Account/shared-mailbox +override (`Private.Account` holds the designer placeholder "Please select +an account.") — eliminating the wrong-mailbox argument variant. Lookup was +by name with no path separators or stray whitespace — eliminating the +stale-folder-ID and path-segment branches. Playbook: +`references/activity-packages/o365-activities/playbooks/mail-folder-not-found.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info/logs carry the full +ArgumentNullException stack; the trace spans carry the `MailFolder` literal +and the scope/auth attributes). + +### Trace spans + +Three spans were recorded for this job and all trace retrieval forms +(`or jobs traces `, `traces spans get --job-key `, and the +session's recorded `traces spans get ` form) replay them: the +RobotJob span (full exception stack with the `GetMailFolder` faulting frame +in Attributes), the Microsoft 365 Scope span, and the faulting "Get Mail +from nonexistent folder" `GetMail` span. In the original session the +`--job-key` form returned "Error retrieving trace ID for job" and the agent +re-fetched by TraceId; the replay serves the genuine spans for every form +so the investigation does not depend on reproducing that transient lookup +failure. + +### Fixture derivations and scrubs + +- `or-jobs-logs-...-level-error.json` is the recorded Error-level log query + (the only logs raw captured); the generic logs rule maps to the same + fixture — the exception row is the only diagnostic content. +- No `--state Faulted` jobs-list raw was recorded, so there is no + Faulted-only fixture; the folder-key rule substring-matches + `--state Faulted` variants and returns the full recorded list. +- No `jobs history` raw was captured; the history fixture is the + Pending → Running → Faulted transition set derived from the recorded job + row's CreationTime/StartTime/EndTime and Id. It confirms a clean single + execution and carries no diagnostic signal beyond `jobs get`. +- Mailbox-content scrub: the span attributes were scanned for real email + subjects/addresses — none were present (the activity faulted at folder + resolution before any message retrieval; `Private.Account` holds the + designer placeholder "Please select an account."), so no content scrub + was needed. Standard scrubs applied: hostname → `MOCK-HOST`, workspace + owner email → `original_email@test.com`, Windows account → + `UIPATH\REPLACEMENT_USER`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs. If the source investigation changes, update the fixture JSONs +directly and re-run the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-getmail-folder-not-found --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/4f3ad42c95.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/4f3ad42c95.json deleted file mode 100644 index 8e13d2b2af..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/4f3ad42c95.json +++ /dev/null @@ -1,460 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "State": "Faulted", - "ProcessName": "TeamMailRetriever", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "CreationTime": "2026-06-11T08:01:50.607Z" - }, - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 32, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/6ac702a3fd.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/6ac702a3fd.json deleted file mode 100644 index d355ec2e44..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/6ac702a3fd.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "6cb078ee427e1cb7", - "TraceId": "e893a4d4e0614cbaa7996ed4de6ee4e2", - "ParentId": null, - "Name": "Robot Job e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "StartTime": "2026-06-11T08:01:56.6824871Z", - "EndTime": "2026-06-11T08:01:59.0662539Z", - "Attributes": "{\"Exception\":\"System.ArgumentNullException: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "cdb279f3-d4a4-4dd2-a68b-1b9670e22b22", - "JobKey": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-11T08:01:59.3671115Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - }, - { - "Id": "340d204ebc836db0", - "TraceId": "e893a4d4e0614cbaa7996ed4de6ee4e2", - "ParentId": "6cb078ee427e1cb7", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-11T08:01:56.8172132Z", - "EndTime": "2026-06-11T08:01:56.8572817Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"PropertiesPanel\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "cdb279f3-d4a4-4dd2-a68b-1b9670e22b22", - "JobKey": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-11T08:01:57.4249586Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - }, - { - "Id": "8f733a5cd9870505", - "TraceId": "e893a4d4e0614cbaa7996ed4de6ee4e2", - "ParentId": "6cb078ee427e1cb7", - "Name": "Get Mail", - "StartTime": "2026-06-11T08:01:57.0150274Z", - "EndTime": "2026-06-11T08:01:57.0160532Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\",\"Private.MailFolder\":\"NoSuchFolder-Repro-123\",\"Public.OnlyUnreadMessages\":\"False\",\"Public.OrderByDate\":\"NewestFirst\",\"Public.MarkAsRead\":\"False\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Get Mail\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "cdb279f3-d4a4-4dd2-a68b-1b9670e22b22", - "JobKey": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-11T08:01:57.4249586Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/9ebd7e8052.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/9ebd7e8052.json deleted file mode 100644 index 702a00e149..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/9ebd7e8052.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:01:59.027Z", - "Message": "Get Mail: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 mail folder not found repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentNullException\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/README.md deleted file mode 100644 index 1b6677dae9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/README.md +++ /dev/null @@ -1,102 +0,0 @@ -# O365 Get Mail — Mail Folder Not Found (unwrapped ArgumentNullException) — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `TeamMailRetriever` (key `e893a4d4-e061-4cba-a799-6ed4de6ee4e2`), -faulted at the legacy **Get Mail** activity ("Get Mail from nonexistent -folder" in `Main.xaml`) inside the Microsoft 365 Scope -with the unwrapped `System.ArgumentNullException: Value cannot be null. -(Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on -this account.')` — the unwrapped form of the mail-folder-not-found -signature; the inner sentence is the match. Faulting frame: -`GraphServiceClientExtensions.GetMailFolder(...)` during folder resolution, -before any message retrieval. The activity's `MailFolder` argument was the -literal name `NoSuchFolder-Repro-123`; folder enumeration of the mailbox -resolved by the scope (interactive-token auth, no shared-mailbox/account -override — the authenticated user's own mailbox) found no match. Fix: -verify/correct the folder name or the `MailFolder` argument (path form for -nested folders), confirm the resolved mailbox, then restart the job. - -### Span evidence pattern - -This scenario's cause disambiguation lives in the **trace spans**: the -"Get Mail" `GetMail` span carries -`Private.MailFolder="NoSuchFolder-Repro-123"` (matching the error verbatim) -and `Public.AuthScopesInvalid="False"` (eliminating the insufficient-scope -branch), and the Microsoft 365 Scope span shows -`AuthenticationType=InteractiveToken` with no Account/shared-mailbox -override (`Private.Account` holds the designer placeholder "Please select -an account.") — eliminating the wrong-mailbox argument variant. Lookup was -by name with no path separators or stray whitespace — eliminating the -stale-folder-ID and path-segment branches. Playbook: -`references/activity-packages/o365-activities/playbooks/mail-folder-not-found.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info/logs carry the full -ArgumentNullException stack; the trace spans carry the `MailFolder` literal -and the scope/auth attributes). - -### Trace spans - -Three spans were recorded for this job and all trace retrieval forms -(`or jobs traces `, `traces spans get --job-key `, and the -session's recorded `traces spans get ` form) replay them: the -RobotJob span (full exception stack with the `GetMailFolder` faulting frame -in Attributes), the Microsoft 365 Scope span, and the faulting "Get Mail -from nonexistent folder" `GetMail` span. In the original session the -`--job-key` form returned "Error retrieving trace ID for job" and the agent -re-fetched by TraceId; the replay serves the genuine spans for every form -so the investigation does not depend on reproducing that transient lookup -failure. - -### Fixture derivations and scrubs - -- `or-jobs-logs-...-level-error.json` is the recorded Error-level log query - (the only logs raw captured); the generic logs rule maps to the same - fixture — the exception row is the only diagnostic content. -- No `--state Faulted` jobs-list raw was recorded, so there is no - Faulted-only fixture; the folder-key rule substring-matches - `--state Faulted` variants and returns the full recorded list. -- No `jobs history` raw was captured; the history fixture is the - Pending → Running → Faulted transition set derived from the recorded job - row's CreationTime/StartTime/EndTime and Id. It confirms a clean single - execution and carries no diagnostic signal beyond `jobs get`. -- Mailbox-content scrub: the span attributes were scanned for real email - subjects/addresses — none were present (the activity faulted at folder - resolution before any message retrieval; `Private.Account` holds the - designer placeholder "Please select an account."), so no content scrub - was needed. Standard scrubs applied: hostname → `MOCK-HOST`, workspace - owner email → `original_email@test.com`, Windows account → - `UIPATH\REPLACEMENT_USER`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs. If the source investigation changes, update the fixture JSONs -directly and re-run the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-getmail-folder-not-found --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/d376cd0548.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/d376cd0548.json deleted file mode 100644 index db2480ab3b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/d376cd0548.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "412283e5-7f06-4ddd-b675-d8d503791de5", - "Info": "Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\r\n\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 mail folder not found repro\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-11T08:01:50.607Z", - "ReleaseName": "TeamMailRetriever", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53475, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T08:01:59.527Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e893a4d4e0614cbaa7996ed4de6ee4e2", - "Id": 4015117 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/f2a0c32aa9.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/f2a0c32aa9.json deleted file mode 100644 index 789a4f7c3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/f2a0c32aa9.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-11T08:01:59.537Z", - "JobId": 4015117 - }, - { - "State": "Running", - "Timestamp": "2026-06-11T08:01:51.030Z", - "JobId": 4015117 - }, - { - "State": "Pending", - "Timestamp": "2026-06-11T08:01:50.657Z", - "JobId": 4015117 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/manifest.json deleted file mode 100644 index 50db126518..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "file": "d376cd0548.json" - }, - { - "match": "or jobs logs e893a4d4-e061-4cba-a799-6ed4de6ee4e2 --level Error", - "file": "9ebd7e8052.json", - "_doc": "Recorded verbatim. The session captured only the Error-level log query; the single row is the job's one Error entry." - }, - { - "match": "or jobs logs e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "file": "9ebd7e8052.json", - "_doc": "No unfiltered-logs raw was recorded; the Error-level set answers any logs variant — the exception row is the only diagnostic content." - }, - { - "match": "or jobs history e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "file": "f2a0c32aa9.json", - "_doc": "No history raw was captured this session; fixture is the Pending -> Running -> Faulted transition set derived from the recorded job row's CreationTime/StartTime/EndTime and Id. Confirms a clean single execution — carries no diagnostic signal beyond jobs get." - }, - { - "match": "or jobs traces e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "file": "6ac702a3fd.json", - "_doc": "Three recorded spans: RobotJob (full System.ArgumentNullException stack with the GetMailFolder faulting frame in Attributes), the Microsoft 365 Scope span (AuthenticationType=InteractiveToken, no Account/shared-mailbox override — Private.Account holds the designer placeholder 'Please select an account.'), and the faulting 'Get Mail' GetMail span with Private.MailFolder='NoSuchFolder-Repro-123' and Public.AuthScopesInvalid='False'. The MailFolder literal + AuthScopesInvalid=False + no account override are the cause-disambiguation evidence (folder absent vs insufficient-scope vs wrong-mailbox)." - }, - { - "match": "traces spans get --job-key e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "file": "6ac702a3fd.json", - "_doc": "Alternate trace-retrieval form (platform traces command). In the original session this form returned 'Error retrieving trace ID for job' and the agent re-fetched by TraceId; the replay serves the genuine spans for every retrieval form so the investigation does not depend on reproducing that transient lookup failure." - }, - { - "match": "traces spans get e893a4d4e0614cbaa7996ed4de6ee4e2", - "file": "6ac702a3fd.json", - "_doc": "The form actually recorded in the session: positional TraceId (32-char hex from the job record) after the --job-key form failed. Same span payload." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "4f3ad42c95.json", - "_doc": "No --state Faulted raw was recorded in this session, so there is no Faulted-only fixture; substring dispatch routes `--state Faulted` variants here too, returning the full recorded list — the failing job is the most recent row." - }, - { - "match": "or jobs list", - "file": "4f3ad42c95.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "min": 1, - "description": "Agent must fetch the failing job to read the unwrapped System.ArgumentNullException — \"Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\" — thrown from GraphServiceClientExtensions.GetMailFolder during legacy GetMail execution." - }, - { - "pattern": "traces", - "min": 1, - "description": "Cause disambiguation: the GetMail span carries Private.MailFolder='NoSuchFolder-Repro-123' and Public.AuthScopesInvalid='False', and the scope span shows InteractiveToken auth with no Account/shared-mailbox override — together pinning 'folder does not exist in the resolved (authenticated user's own) mailbox' over the insufficient-scope and wrong-mailbox branches." - }, - { - "pattern": "or jobs logs e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..3acd04ec98 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:01:59.027Z\",\r\n \"Message\": \"Get Mail: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 mail folder not found repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "e893a4d4-e061-4cba-a799-6ed4de6ee4e2" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"6cb078ee427e1cb7\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"StartTime\": \"2026-06-11T08:01:56.6824871Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.0662539Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentNullException: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\\\r\\\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:59.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"340d204ebc836db0\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": \"6cb078ee427e1cb7\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-11T08:01:56.8172132Z\",\r\n \"EndTime\": \"2026-06-11T08:01:56.8572817Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:57.4249586Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"8f733a5cd9870505\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": \"6cb078ee427e1cb7\",\r\n \"Name\": \"Get Mail\",\r\n \"StartTime\": \"2026-06-11T08:01:57.0150274Z\",\r\n \"EndTime\": \"2026-06-11T08:01:57.0160532Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"NoSuchFolder-Repro-123\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:57.4249586Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 32,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e893a4d4-e061-4cba-a799-6ed4de6ee4e2" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"412283e5-7f06-4ddd-b675-d8d503791de5\",\r\n \"Info\": \"Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 mail folder not found repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\",\r\n \"ReleaseName\": \"TeamMailRetriever\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53475,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"Id\": 4015117\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e893a4d4-e061-4cba-a799-6ed4de6ee4e2" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:01:59.027Z\",\r\n \"Message\": \"Get Mail: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 mail folder not found repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "e893a4d4-e061-4cba-a799-6ed4de6ee4e2" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-11T08:01:59.537Z\",\r\n \"JobId\": 4015117\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-11T08:01:51.030Z\",\r\n \"JobId\": 4015117\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-11T08:01:50.657Z\",\r\n \"JobId\": 4015117\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "e893a4d4-e061-4cba-a799-6ed4de6ee4e2" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"6cb078ee427e1cb7\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"StartTime\": \"2026-06-11T08:01:56.6824871Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.0662539Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentNullException: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\\\r\\\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:59.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"340d204ebc836db0\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": \"6cb078ee427e1cb7\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-11T08:01:56.8172132Z\",\r\n \"EndTime\": \"2026-06-11T08:01:56.8572817Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:57.4249586Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"8f733a5cd9870505\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": \"6cb078ee427e1cb7\",\r\n \"Name\": \"Get Mail\",\r\n \"StartTime\": \"2026-06-11T08:01:57.0150274Z\",\r\n \"EndTime\": \"2026-06-11T08:01:57.0160532Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"NoSuchFolder-Repro-123\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:57.4249586Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "e893a4d4e0614cbaa7996ed4de6ee4e2" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"6cb078ee427e1cb7\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"StartTime\": \"2026-06-11T08:01:56.6824871Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.0662539Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentNullException: Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\\\\r\\\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailFolder(IUserRequestBuilder account, String folderPath, Boolean createIfMissing, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:59.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"340d204ebc836db0\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": \"6cb078ee427e1cb7\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-11T08:01:56.8172132Z\",\r\n \"EndTime\": \"2026-06-11T08:01:56.8572817Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:57.4249586Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"8f733a5cd9870505\",\r\n \"TraceId\": \"e893a4d4e0614cbaa7996ed4de6ee4e2\",\r\n \"ParentId\": \"6cb078ee427e1cb7\",\r\n \"Name\": \"Get Mail\",\r\n \"StartTime\": \"2026-06-11T08:01:57.0150274Z\",\r\n \"EndTime\": \"2026-06-11T08:01:57.0160532Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"NoSuchFolder-Repro-123\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"cdb279f3-d4a4-4dd2-a68b-1b9670e22b22\",\r\n \"JobKey\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-11T08:01:57.4249586Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 32,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get e893a4d4-e061-4cba-a799-6ed4de6ee4e2", + "min": 1, + "description": "Agent must fetch the failing job to read the unwrapped System.ArgumentNullException — \"Value cannot be null. (Parameter 'Folder named 'NoSuchFolder-Repro-123' could not be found on this account.')\" — thrown from GraphServiceClientExtensions.GetMailFolder during legacy GetMail execution." + }, + { + "pattern": "traces", + "min": 1, + "description": "Cause disambiguation: the GetMail span carries Private.MailFolder='NoSuchFolder-Repro-123' and Public.AuthScopesInvalid='False', and the scope span shows InteractiveToken auth with no Account/shared-mailbox override — together pinning 'folder does not exist in the resolved (authenticated user's own) mailbox' over the insufficient-scope and wrong-mailbox branches." + }, + { + "pattern": "or jobs logs e893a4d4-e061-4cba-a799-6ed4de6ee4e2", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history e893a4d4-e061-4cba-a799-6ed4de6ee4e2", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/task.yaml index d910cc0fa2..3ba1b12b98 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-folder-not-found/task.yaml @@ -26,21 +26,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/README.md new file mode 100644 index 0000000000..5dad755417 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/README.md @@ -0,0 +1,62 @@ +# O365 Get Mail Invalid OData Query — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `InboxQueryRunner` (key `4b4a58f8-6a7c-4af2-8700-1b6479b51028`), +faulted with a raw `Microsoft.Graph.ServiceException` `Code: BadRequest` — +`Invalid filter clause: ')' or operator expected at position 56 in +'(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and +unread is true)'` — thrown by the legacy O365 `GetMail` activity. The +activity prepends the date-range clause itself; the malformed segment is the +user-configured `Query`, which uses natural-language operators (`equals` is +not an OData operator, `unread` is not a Graph message property). Graph +rejected the request at parse time — deterministic, independent of mailbox +content, not auth/throttling/transient. Fix: correct the `Query` to valid +OData (`subject eq 'invoice'`) and/or use the activity's built-in Only Unread +Messages option. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info/logs carry the full fault stack and +the composed `$filter` echoed verbatim). + +### Trace spans + +No trace spans were recorded for this job in the original session. Both trace +retrieval forms (`or jobs traces ` and `traces spans get --job-key +`) are mocked with a well-formed empty `JobTraces` response so the agent +sees "no spans" rather than an unmocked default. The root cause is fully +visible in `jobs get` / `jobs logs`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs (the generator's transcript pass is unreliable for this session). If +the source investigation changes, update the fixture JSONs directly and re-run +the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-getmail-invalid-odata-query --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/6688b9dbb0.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/6688b9dbb0.json deleted file mode 100644 index 5228608ef3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/6688b9dbb0.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T18:37:25.822Z", - "Message": "Get Mail: Code: BadRequest\r\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\r\nInner error:\r\n\tAdditionalData:\r\n\tdate: 2026-06-10T18:37:25\r\n\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\r\n\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\r\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\r\n\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Mail invalid OData query repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nMicrosoft.Graph.ServiceException\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.MailFolderMessagesCollectionRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetAllMessagesAsync(IUserRequestBuilder account, MailFolder folder, Boolean includeSubfolders, Nullable`1 useSearchApi, List`1 headers, String query, String orderBy, Nullable`1 top, Int32 count, CancellationToken cancellationToken)\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/6856bb397e.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/6856bb397e.json deleted file mode 100644 index 6448b1c421..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/6856bb397e.json +++ /dev/null @@ -1,264 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 18, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/91e16d96ef.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/91e16d96ef.json deleted file mode 100644 index 65b107e0d3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/91e16d96ef.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "3546d29b-d277-4aac-83f8-cb3bf344b6c0", - "Info": "Code: BadRequest\r\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\r\nInner error:\r\n\tAdditionalData:\r\n\tdate: 2026-06-10T18:37:25\r\n\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\r\n\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\r\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\r\n\r\n\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Mail invalid OData query repro\"\r\n at Main \"Main\"\r\n\r\nMicrosoft.Graph.ServiceException: Code: BadRequest\r\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\r\nInner error:\r\n\tAdditionalData:\r\n\tdate: 2026-06-10T18:37:25\r\n\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\r\n\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\r\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\r\n\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.MailFolderMessagesCollectionRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetAllMessagesAsync(IUserRequestBuilder account, MailFolder folder, Boolean includeSubfolders, Nullable`1 useSearchApi, List`1 headers, String query, String orderBy, Nullable`1 top, Int32 count, CancellationToken cancellationToken)\r\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T18:37:22.647Z", - "ReleaseName": "InboxQueryRunner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53470, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T18:37:26.293Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "4b4a58f86a7c4af287001b6479b51028", - "Id": 4015111 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/README.md deleted file mode 100644 index 8d39c82567..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/README.md +++ /dev/null @@ -1,61 +0,0 @@ -# O365 Get Mail Invalid OData Query — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `InboxQueryRunner` (key `4b4a58f8-6a7c-4af2-8700-1b6479b51028`), -faulted with a raw `Microsoft.Graph.ServiceException` `Code: BadRequest` — -`Invalid filter clause: ')' or operator expected at position 56 in -'(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and -unread is true)'` — thrown by the legacy O365 `GetMail` activity. The -activity prepends the date-range clause itself; the malformed segment is the -user-configured `Query`, which uses natural-language operators (`equals` is -not an OData operator, `unread` is not a Graph message property). Graph -rejected the request at parse time — deterministic, independent of mailbox -content, not auth/throttling/transient. Fix: correct the `Query` to valid -OData (`subject eq 'invoice'`) and/or use the activity's built-in Only Unread -Messages option. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info/logs carry the full fault stack and -the composed `$filter` echoed verbatim). - -### Trace spans - -No trace spans were recorded for this job in the original session. Both trace -retrieval forms (`or jobs traces ` and `traces spans get --job-key -`) are mocked with a well-formed empty `JobTraces` response so the agent -sees "no spans" rather than an unmocked default. The root cause is fully -visible in `jobs get` / `jobs logs`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs (the generator's transcript pass is unreliable for this session). If -the source investigation changes, update the fixture JSONs directly and re-run -the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-getmail-invalid-odata-query --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/b11b041211.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/b11b041211.json deleted file mode 100644 index 9236dc9c28..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/b11b041211.json +++ /dev/null @@ -1,390 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 27, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/d0d946603e.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/d0d946603e.json deleted file mode 100644 index 185f896639..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/d0d946603e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T18:37:26.303Z", - "JobId": 4015111 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T18:37:23.017Z", - "JobId": 4015111 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T18:37:22.697Z", - "JobId": 4015111 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/manifest.json deleted file mode 100644 index 2bc56664ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/m/r/manifest.json +++ /dev/null @@ -1,92 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "file": "91e16d96ef.json" - }, - { - "match": "or jobs logs 4b4a58f8-6a7c-4af2-8700-1b6479b51028 --level Error", - "file": "6688b9dbb0.json" - }, - { - "match": "or jobs logs 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "file": "6688b9dbb0.json" - }, - { - "match": "or jobs history 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "file": "d0d946603e.json" - }, - { - "match": "or jobs traces 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "file": "2d930e2440.json", - "_doc": "No trace spans were recorded for this job — the fixture is a well-formed empty JobTraces response so the agent reads 'no spans' instead of misreading a bare []. The root cause is fully visible in jobs get / jobs logs." - }, - { - "match": "traces spans get --job-key 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "file": "2d930e2440.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "6856bb397e.json", - "_doc": "Faulted-only subset of the recorded folder job list (rows verbatim, filtered to State == Faulted)." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "b11b041211.json" - }, - { - "match": "or jobs list", - "file": "b11b041211.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "min": 1, - "description": "Agent must fetch the failing job to read the raw Microsoft.Graph.ServiceException Code: BadRequest — Invalid filter clause ... in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)' from the legacy GetMail activity." - }, - { - "pattern": "or jobs logs 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "traces", - "min": 0, - "description": "OPTIONAL — no spans were recorded for this job; the empty response adds nothing. Root cause is already fully visible in jobs get." - }, - { - "pattern": "or jobs history 4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/uip-fixture.json new file mode 100644 index 0000000000..da73530699 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/data/uip-fixture.json @@ -0,0 +1,156 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 18,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4b4a58f8-6a7c-4af2-8700-1b6479b51028", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T18:37:25.822Z\",\r\n \"Message\": \"Get Mail: Code: BadRequest\\r\\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\\r\\nInner error:\\r\\n\\tAdditionalData:\\r\\n\\tdate: 2026-06-10T18:37:25\\r\\n\\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Mail invalid OData query repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nMicrosoft.Graph.ServiceException\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MailFolderMessagesCollectionRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetAllMessagesAsync(IUserRequestBuilder account, MailFolder folder, Boolean includeSubfolders, Nullable`1 useSearchApi, List`1 headers, String query, String orderBy, Nullable`1 top, Int32 count, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "4b4a58f8-6a7c-4af2-8700-1b6479b51028" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 27,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "4b4a58f8-6a7c-4af2-8700-1b6479b51028" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"3546d29b-d277-4aac-83f8-cb3bf344b6c0\",\r\n \"Info\": \"Code: BadRequest\\r\\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\\r\\nInner error:\\r\\n\\tAdditionalData:\\r\\n\\tdate: 2026-06-10T18:37:25\\r\\n\\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Mail invalid OData query repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nMicrosoft.Graph.ServiceException: Code: BadRequest\\r\\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\\r\\nInner error:\\r\\n\\tAdditionalData:\\r\\n\\tdate: 2026-06-10T18:37:25\\r\\n\\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MailFolderMessagesCollectionRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetAllMessagesAsync(IUserRequestBuilder account, MailFolder folder, Boolean includeSubfolders, Nullable`1 useSearchApi, List`1 headers, String query, String orderBy, Nullable`1 top, Int32 count, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\",\r\n \"ReleaseName\": \"InboxQueryRunner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53470,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"4b4a58f86a7c4af287001b6479b51028\",\r\n \"Id\": 4015111\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4b4a58f8-6a7c-4af2-8700-1b6479b51028" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T18:37:25.822Z\",\r\n \"Message\": \"Get Mail: Code: BadRequest\\r\\nMessage: Invalid filter clause: ')' or operator expected at position 56 in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)'.\\r\\nInner error:\\r\\n\\tAdditionalData:\\r\\n\\tdate: 2026-06-10T18:37:25\\r\\n\\trequest-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\tclient-request-id: 222b6c61-240d-4608-bff0-1ee696503189\\r\\nClientRequestId: 222b6c61-240d-4608-bff0-1ee696503189\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Mail invalid OData query repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nMicrosoft.Graph.ServiceException\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MailFolderMessagesCollectionRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetAllMessagesAsync(IUserRequestBuilder account, MailFolder folder, Boolean includeSubfolders, Nullable`1 useSearchApi, List`1 headers, String query, String orderBy, Nullable`1 top, Int32 count, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Services.Graph.Mail.Extensions.GraphServiceClientExtensions.GetMailAsync(GraphServiceClient graphClient, Boolean getAsHTML, String query, Nullable`1 top, String address, String mailFolder, Boolean includeSubfolders, Nullable`1 useSearchApi, String orderBy, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "4b4a58f8-6a7c-4af2-8700-1b6479b51028" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T18:37:26.303Z\",\r\n \"JobId\": 4015111\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T18:37:23.017Z\",\r\n \"JobId\": 4015111\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T18:37:22.697Z\",\r\n \"JobId\": 4015111\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "4b4a58f8-6a7c-4af2-8700-1b6479b51028" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 27,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 4b4a58f8-6a7c-4af2-8700-1b6479b51028", + "min": 1, + "description": "Agent must fetch the failing job to read the raw Microsoft.Graph.ServiceException Code: BadRequest — Invalid filter clause ... in '(receivedDateTime ge 1900-01-01T00:00:00Z) and (subject equals 'invoice' and unread is true)' from the legacy GetMail activity." + }, + { + "pattern": "or jobs logs 4b4a58f8-6a7c-4af2-8700-1b6479b51028", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "traces", + "min": 0, + "description": "OPTIONAL — no spans were recorded for this job; the empty response adds nothing. Root cause is already fully visible in jobs get." + }, + { + "pattern": "or jobs history 4b4a58f8-6a7c-4af2-8700-1b6479b51028", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/task.yaml index cdf46f36b9..70de62de6c 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-invalid-odata-query/task.yaml @@ -20,21 +20,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/README.md new file mode 100644 index 0000000000..3eab9a8885 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/README.md @@ -0,0 +1,95 @@ +# O365 Get Mail — Stale Message ID (ErrorItemNotFound) — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `SupportMailLookup` (key `cbbaaaf6-b416-4025-9a98-c8ee50990f76`), +faulted at the **Get Mail** activity ("Get Mail by corrupted ID (expect +ErrorInvalidIdMalformed; for ErrorItemNotFound use the ID of a deleted +email)" in `Main.xaml`) inside the Microsoft 365 Scope +with a raw `Microsoft.Graph.ServiceException` — `Code: ErrorItemNotFound`, +`Message: The specified object was not found in the store., The process +failed to get the correct properties.`, `Status Code: NotFound`. The raw +(unwrapped) ServiceException marks the legacy O365 activity, not a +Connections-based one. The activity's `EmailId` was bound to variable +`staleId` — an ID that passes Outlook's integrity check (so not +`ErrorInvalidIdMalformed`) but resolves to no message in the store: the +signature of a stale/deleted message ID. Fix: confirm the message still +exists / fix the ID source, and re-fetch by filter at consumption time +instead of persisting message IDs across runs. + +### Span evidence pattern + +This scenario's distinguishing evidence lives in the **trace spans**, not +the job Info alone: the recorded spans show a two-`GetMail` sequence inside +the same Microsoft 365 Scope — "Get newest email (source of a real ID)" +**succeeded** against the same Inbox/connection immediately before the +faulting "Get Mail by corrupted ID ..." span whose attributes carry +`Private.EmailId="staleId"`. The preceding success on the same mailbox +eliminates the playbook's sibling branches: mailbox mismatch +(`ErrorInvalidMailboxItemId` absent), authentication, and +missing-scope-as-404 causes. Playbook: +`references/activity-packages/o365-activities/playbooks/mail-message-not-found.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info/logs carry the full ServiceException +stack; the trace spans carry the `staleId` binding and the +preceding-success evidence). + +### Trace spans + +Four spans were recorded for this job and both trace retrieval forms +(`or jobs traces ` and `traces spans get --job-key `) replay them: +the RobotJob span (full exception stack incl. `Status Code: NotFound` in +Attributes), the Microsoft 365 Scope span, the successful "Get newest email +(source of a real ID)" `GetMail` span, and the faulting "Get Mail by +corrupted ID ..." `GetMail` span (`Private.EmailId="staleId"`). + +### Fixture derivations and scrubs + +- `or-jobs-logs-...-level-error.json` is the Error-level subset of the + recorded log set (rows verbatim, filtered to `Level == "Error"`). +- No `--state Faulted` jobs-list raw was recorded, so there is no + Faulted-only fixture; the folder-key rule substring-matches + `--state Faulted` variants and returns the full recorded list. +- No `is connections` probe was run in this session, so the manifest has no + IS rule — unmatched commands fall to the empty `unmocked_default`. +- Mailbox-content scrub: the span attributes were scanned for real email + subjects/addresses (the first `GetMail` fetched the newest real email) — + none were present in the recorded attributes (`Private.Results` holds the + variable name `results`, not message content), so no content scrub was + needed. Standard scrubs applied: hostname → `MOCK-HOST`, workspace owner + email → `original_email@test.com`, Windows account → + `UIPATH\REPLACEMENT_USER`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs. If the source investigation changes, update the fixture JSONs +directly and re-run the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-getmail-message-not-found --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/2543de7fe3.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/2543de7fe3.json deleted file mode 100644 index 22754251af..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/2543de7fe3.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-10T19:45:17.783Z", - "Message": "SupportMailLookup execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T19:45:17.173Z", - "Message": "Get Mail: Code: ErrorItemNotFound\r\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\r\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\r\n\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Mail message not found repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nMicrosoft.Graph.ServiceException\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-10T19:45:14.100Z", - "Message": "SupportMailLookup execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/321c404114.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/321c404114.json deleted file mode 100644 index b2e6ab487c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/321c404114.json +++ /dev/null @@ -1,110 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-3a8d-b2891c40f816", - "TraceId": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ParentId": null, - "Name": "Robot Job cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "StartTime": "2026-06-10T19:45:14.0765148Z", - "EndTime": "2026-06-10T19:45:17.204113Z", - "Attributes": "{\"Exception\":\"Status Code: NotFound\\r\\nMicrosoft.Graph.ServiceException: Code: ErrorItemNotFound\\r\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\r\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\r\\n\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "18539c8d-ba23-4bae-8c29-dec056d789ae", - "JobKey": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:45:17.7503705Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-9f56-fec8f6a36b24", - "TraceId": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ParentId": "00000000-0000-0000-3a8d-b2891c40f816", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-10T19:45:14.2066492Z", - "EndTime": "2026-06-10T19:45:14.2403156Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"PropertiesPanel\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "18539c8d-ba23-4bae-8c29-dec056d789ae", - "JobKey": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:45:15.3760208Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-af04-f23d16fc975e", - "TraceId": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ParentId": "00000000-0000-0000-3a8d-b2891c40f816", - "Name": "Get newest email (source of a real ID)", - "StartTime": "2026-06-10T19:45:14.5492848Z", - "EndTime": "2026-06-10T19:45:14.5495956Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\",\"Private.MailFolder\":\"Inbox\",\"Public.Top\":\"1\",\"Public.OnlyUnreadMessages\":\"False\",\"Public.OrderByDate\":\"NewestFirst\",\"Public.MarkAsRead\":\"False\",\"Private.Results\":\"results\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Get newest email (source of a real ID)\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "18539c8d-ba23-4bae-8c29-dec056d789ae", - "JobKey": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:45:15.3760208Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-589a-055ee001e3d7", - "TraceId": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ParentId": "00000000-0000-0000-3a8d-b2891c40f816", - "Name": "Get Mail", - "StartTime": "2026-06-10T19:45:17.0062526Z", - "EndTime": "2026-06-10T19:45:17.0063246Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\",\"Private.MailFolder\":\"Inbox\",\"Private.EmailId\":\"staleId\",\"Public.OnlyUnreadMessages\":\"False\",\"Public.OrderByDate\":\"NewestFirst\",\"Public.MarkAsRead\":\"False\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Get Mail\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "18539c8d-ba23-4bae-8c29-dec056d789ae", - "JobKey": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:45:17.2659757Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/371289d092.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/371289d092.json deleted file mode 100644 index 37636366bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/371289d092.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T19:45:17.173Z", - "Message": "Get Mail: Code: ErrorItemNotFound\r\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\r\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\r\n\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Mail message not found repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nMicrosoft.Graph.ServiceException\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/383a6311e5.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/383a6311e5.json deleted file mode 100644 index ae5c5f80cc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/383a6311e5.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T19:45:17.910Z", - "JobId": 4015115 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T19:45:13.697Z", - "JobId": 4015115 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T19:45:12.923Z", - "JobId": 4015115 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/9cb54f2d22.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/9cb54f2d22.json deleted file mode 100644 index 844d3f06a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/9cb54f2d22.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "75b4ed96-5dea-4a5b-b1a9-b930ced7e21b", - "Info": "Code: ErrorItemNotFound\r\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\r\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\r\n\r\n\r\nin Main.xaml\r\n at GetMail \"Get Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Mail message not found repro\"\r\n at Main \"Main\"\r\n\r\nMicrosoft.Graph.ServiceException: Code: ErrorItemNotFound\r\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\r\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\r\n\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T19:45:12.887Z", - "ReleaseName": "SupportMailLookup", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53474, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T19:45:17.903Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "cbbaaaf6b41640259a98c8ee50990f76", - "Id": 4015115 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/README.md deleted file mode 100644 index 2a4d94a4b1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/README.md +++ /dev/null @@ -1,94 +0,0 @@ -# O365 Get Mail — Stale Message ID (ErrorItemNotFound) — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `SupportMailLookup` (key `cbbaaaf6-b416-4025-9a98-c8ee50990f76`), -faulted at the **Get Mail** activity ("Get Mail by corrupted ID (expect -ErrorInvalidIdMalformed; for ErrorItemNotFound use the ID of a deleted -email)" in `Main.xaml`) inside the Microsoft 365 Scope -with a raw `Microsoft.Graph.ServiceException` — `Code: ErrorItemNotFound`, -`Message: The specified object was not found in the store., The process -failed to get the correct properties.`, `Status Code: NotFound`. The raw -(unwrapped) ServiceException marks the legacy O365 activity, not a -Connections-based one. The activity's `EmailId` was bound to variable -`staleId` — an ID that passes Outlook's integrity check (so not -`ErrorInvalidIdMalformed`) but resolves to no message in the store: the -signature of a stale/deleted message ID. Fix: confirm the message still -exists / fix the ID source, and re-fetch by filter at consumption time -instead of persisting message IDs across runs. - -### Span evidence pattern - -This scenario's distinguishing evidence lives in the **trace spans**, not -the job Info alone: the recorded spans show a two-`GetMail` sequence inside -the same Microsoft 365 Scope — "Get newest email (source of a real ID)" -**succeeded** against the same Inbox/connection immediately before the -faulting "Get Mail by corrupted ID ..." span whose attributes carry -`Private.EmailId="staleId"`. The preceding success on the same mailbox -eliminates the playbook's sibling branches: mailbox mismatch -(`ErrorInvalidMailboxItemId` absent), authentication, and -missing-scope-as-404 causes. Playbook: -`references/activity-packages/o365-activities/playbooks/mail-message-not-found.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info/logs carry the full ServiceException -stack; the trace spans carry the `staleId` binding and the -preceding-success evidence). - -### Trace spans - -Four spans were recorded for this job and both trace retrieval forms -(`or jobs traces ` and `traces spans get --job-key `) replay them: -the RobotJob span (full exception stack incl. `Status Code: NotFound` in -Attributes), the Microsoft 365 Scope span, the successful "Get newest email -(source of a real ID)" `GetMail` span, and the faulting "Get Mail by -corrupted ID ..." `GetMail` span (`Private.EmailId="staleId"`). - -### Fixture derivations and scrubs - -- `or-jobs-logs-...-level-error.json` is the Error-level subset of the - recorded log set (rows verbatim, filtered to `Level == "Error"`). -- No `--state Faulted` jobs-list raw was recorded, so there is no - Faulted-only fixture; the folder-key rule substring-matches - `--state Faulted` variants and returns the full recorded list. -- No `is connections` probe was run in this session, so the manifest has no - IS rule — unmatched commands fall to the empty `unmocked_default`. -- Mailbox-content scrub: the span attributes were scanned for real email - subjects/addresses (the first `GetMail` fetched the newest real email) — - none were present in the recorded attributes (`Private.Results` holds the - variable name `results`, not message content), so no content scrub was - needed. Standard scrubs applied: hostname → `MOCK-HOST`, workspace owner - email → `original_email@test.com`, Windows account → - `UIPATH\REPLACEMENT_USER`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs. If the source investigation changes, update the fixture JSONs -directly and re-run the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-getmail-message-not-found --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/de25ba3ea5.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/de25ba3ea5.json deleted file mode 100644 index 467ca1f908..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/de25ba3ea5.json +++ /dev/null @@ -1,446 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 31, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/manifest.json deleted file mode 100644 index e04c5d3a65..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,90 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "file": "9cb54f2d22.json" - }, - { - "match": "or jobs logs cbbaaaf6-b416-4025-9a98-c8ee50990f76 --level Error", - "file": "371289d092.json", - "_doc": "Error-level subset of the recorded log set (rows verbatim, filtered to Level == Error)." - }, - { - "match": "or jobs logs cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "file": "2543de7fe3.json", - "_doc": "Full log set (execution started -> Error -> execution ended) for unfiltered log retrieval." - }, - { - "match": "or jobs history cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "file": "383a6311e5.json" - }, - { - "match": "or jobs traces cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "file": "321c404114.json", - "_doc": "Four recorded spans: RobotJob (full Microsoft.Graph.ServiceException ErrorItemNotFound stack incl. 'Status Code: NotFound' in Attributes), the Microsoft 365 Scope span, the preceding 'Get newest email (source of a real ID)' GetMail span that SUCCEEDED on the same Inbox, and the faulting 'Get Mail by corrupted ID ...' GetMail span with Private.EmailId=\"staleId\". The two-GetMail sequence and the staleId binding are the core evidence for the stale/deleted-ID diagnosis." - }, - { - "match": "traces spans get --job-key cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "file": "321c404114.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same span payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "de25ba3ea5.json", - "_doc": "No --state Faulted raw was recorded in this session, so there is no Faulted-only fixture; substring dispatch routes `--state Faulted` variants here too, returning the full recorded list — the failing job is the most recent row." - }, - { - "match": "or jobs list", - "file": "de25ba3ea5.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "min": 1, - "description": "Agent must fetch the failing job to read the raw Microsoft.Graph.ServiceException — Code: ErrorItemNotFound, 'The specified object was not found in the store., The process failed to get the correct properties.' — from the legacy GetMail activity." - }, - { - "pattern": "traces", - "min": 1, - "description": "Core evidence: the spans carry the two-GetMail sequence — 'Get newest email (source of a real ID)' SUCCEEDED on the same Inbox right before the faulting 'Get Mail by corrupted ID ...' span whose Private.EmailId=\"staleId\" — distinguishing a stale/deleted message ID from mailbox-mismatch (ErrorInvalidMailboxItemId), auth, and scope-as-404 causes." - }, - { - "pattern": "or jobs logs cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..e280da5845 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/data/uip-fixture.json @@ -0,0 +1,142 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "cbbaaaf6-b416-4025-9a98-c8ee50990f76", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T19:45:17.173Z\",\r\n \"Message\": \"Get Mail: Code: ErrorItemNotFound\\r\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\r\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Mail message not found repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nMicrosoft.Graph.ServiceException\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "cbbaaaf6-b416-4025-9a98-c8ee50990f76" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"StartTime\": \"2026-06-10T19:45:14.0765148Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.204113Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"Status Code: NotFound\\\\r\\\\nMicrosoft.Graph.ServiceException: Code: ErrorItemNotFound\\\\r\\\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\\\r\\\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\\\r\\\\n\\\\r\\\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\\\r\\\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\\\r\\\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\\\r\\\\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:17.7503705Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9f56-fec8f6a36b24\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T19:45:14.2066492Z\",\r\n \"EndTime\": \"2026-06-10T19:45:14.2403156Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:15.3760208Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-af04-f23d16fc975e\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"Name\": \"Get newest email (source of a real ID)\",\r\n \"StartTime\": \"2026-06-10T19:45:14.5492848Z\",\r\n \"EndTime\": \"2026-06-10T19:45:14.5495956Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"Inbox\\\",\\\"Public.Top\\\":\\\"1\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Private.Results\\\":\\\"results\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get newest email (source of a real ID)\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:15.3760208Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-589a-055ee001e3d7\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"Name\": \"Get Mail\",\r\n \"StartTime\": \"2026-06-10T19:45:17.0062526Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.0063246Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"Inbox\\\",\\\"Private.EmailId\\\":\\\"staleId\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:17.2659757Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 31,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cbbaaaf6-b416-4025-9a98-c8ee50990f76" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"75b4ed96-5dea-4a5b-b1a9-b930ced7e21b\",\r\n \"Info\": \"Code: ErrorItemNotFound\\r\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\r\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\r\\n\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Mail message not found repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nMicrosoft.Graph.ServiceException: Code: ErrorItemNotFound\\r\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\r\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\r\\n\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\",\r\n \"ReleaseName\": \"SupportMailLookup\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53474,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"cbbaaaf6b41640259a98c8ee50990f76\",\r\n \"Id\": 4015115\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cbbaaaf6-b416-4025-9a98-c8ee50990f76" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T19:45:17.783Z\",\r\n \"Message\": \"SupportMailLookup execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T19:45:17.173Z\",\r\n \"Message\": \"Get Mail: Code: ErrorItemNotFound\\r\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\r\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\r\\n\\r\\nin Main.xaml\\r\\n at GetMail \\\"Get Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Mail message not found repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nMicrosoft.Graph.ServiceException\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T19:45:14.100Z\",\r\n \"Message\": \"SupportMailLookup execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cbbaaaf6-b416-4025-9a98-c8ee50990f76" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T19:45:17.910Z\",\r\n \"JobId\": 4015115\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T19:45:13.697Z\",\r\n \"JobId\": 4015115\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T19:45:12.923Z\",\r\n \"JobId\": 4015115\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "cbbaaaf6-b416-4025-9a98-c8ee50990f76" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"StartTime\": \"2026-06-10T19:45:14.0765148Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.204113Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"Status Code: NotFound\\\\r\\\\nMicrosoft.Graph.ServiceException: Code: ErrorItemNotFound\\\\r\\\\nMessage: The specified object was not found in the store., The process failed to get the correct properties.\\\\r\\\\nClientRequestId: fe5c641f-dd85-4340-acd1-aad659b9dc43\\\\r\\\\n\\\\r\\\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\\\r\\\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\\\r\\\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\\\r\\\\n at Microsoft.Graph.MessageRequest.GetAsync(CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailAsync(GetMailRequest request, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:17.7503705Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-9f56-fec8f6a36b24\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T19:45:14.2066492Z\",\r\n \"EndTime\": \"2026-06-10T19:45:14.2403156Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:15.3760208Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-af04-f23d16fc975e\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"Name\": \"Get newest email (source of a real ID)\",\r\n \"StartTime\": \"2026-06-10T19:45:14.5492848Z\",\r\n \"EndTime\": \"2026-06-10T19:45:14.5495956Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"Inbox\\\",\\\"Public.Top\\\":\\\"1\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Private.Results\\\":\\\"results\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get newest email (source of a real ID)\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:15.3760208Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-589a-055ee001e3d7\",\r\n \"TraceId\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ParentId\": \"00000000-0000-0000-3a8d-b2891c40f816\",\r\n \"Name\": \"Get Mail\",\r\n \"StartTime\": \"2026-06-10T19:45:17.0062526Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.0063246Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetMail\\\",\\\"Private.MailFolder\\\":\\\"Inbox\\\",\\\"Private.EmailId\\\":\\\"staleId\\\",\\\"Public.OnlyUnreadMessages\\\":\\\"False\\\",\\\"Public.OrderByDate\\\":\\\"NewestFirst\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Get Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"18539c8d-ba23-4bae-8c29-dec056d789ae\",\r\n \"JobKey\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:45:17.2659757Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 31,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get cbbaaaf6-b416-4025-9a98-c8ee50990f76", + "min": 1, + "description": "Agent must fetch the failing job to read the raw Microsoft.Graph.ServiceException — Code: ErrorItemNotFound, 'The specified object was not found in the store., The process failed to get the correct properties.' — from the legacy GetMail activity." + }, + { + "pattern": "traces", + "min": 1, + "description": "Core evidence: the spans carry the two-GetMail sequence — 'Get newest email (source of a real ID)' SUCCEEDED on the same Inbox right before the faulting 'Get Mail by corrupted ID ...' span whose Private.EmailId=\"staleId\" — distinguishing a stale/deleted message ID from mailbox-mismatch (ErrorInvalidMailboxItemId), auth, and scope-as-404 causes." + }, + { + "pattern": "or jobs logs cbbaaaf6-b416-4025-9a98-c8ee50990f76", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history cbbaaaf6-b416-4025-9a98-c8ee50990f76", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/task.yaml index 3689d62837..0ba87a092a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-getmail-message-not-found/task.yaml @@ -22,21 +22,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/README.md new file mode 100644 index 0000000000..84336df4e7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/README.md @@ -0,0 +1,36 @@ +# O365 Interactive Token Timeout — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The Microsoft 365 Scope in the **ERN** process is configured for **Interactive Token** authentication (no account bound — the Account field still reads "Please select an account.", Integration Service off), which is the wrong and fragile auth choice for this Agent/StudioPro execution context. Interactive Token requires a human to complete a browser sign-in within a 30-second window; in this no-bound-account agent run nobody completed it, so authentication timed out and the job faulted. Switch the scope to app-only authentication (Application ID + Client Secret, or Application ID + Certificate). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name o365-interactive-token-timeout --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/15736f592f.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/15736f592f.json deleted file mode 100644 index 056ec6f135..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/15736f592f.json +++ /dev/null @@ -1,306 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 21, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/99d0038951.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/99d0038951.json deleted file mode 100644 index e7dd7dc7a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/99d0038951.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "Id": "00000000-0000-0000-0000-000000000001", - "TraceId": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "ParentId": null, - "Name": "Robot Job b7974725-4438-4315-a9cc-9ec41f8acd62", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "Attributes": "{\"Exception\":\"System.TimeoutException: The client did not complete the authentication after 30 seconds, and as a result the operation was canceled. Authentication type: InteractiveToken.\\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.LoginWithInteractiveTokenAsync()\\n at UiPath.Shared.Activities.Utilities.TaskExtensions.TimeoutAfter()\"}", - "Status": 2, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "SpanType": "RobotJob", - "JobKey": "b7974725-4438-4315-a9cc-9ec41f8acd62" - }, - { - "Id": "00000000-0000-0000-0000-000000000002", - "TraceId": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "ParentId": "00000000-0000-0000-0000-000000000001", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-09T10:49:39.313Z", - "EndTime": "2026-06-09T10:49:39.351Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "SpanType": "Activity", - "JobKey": "b7974725-4438-4315-a9cc-9ec41f8acd62" - }, - { - "Id": "00000000-0000-0000-0000-000000000003", - "TraceId": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "ParentId": "00000000-0000-0000-0000-000000000001", - "Name": "Get Email List", - "StartTime": "2026-06-09T10:49:39.921Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "Attributes": "{\"Exception\":\"System.TimeoutException: The client did not complete the authentication after 30 seconds, and as a result the operation was canceled. Authentication type: InteractiveToken.\",\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.GetEmailListConnections\",\"Private.MailFolderArg\":{\"FolderName\":\"Inbox\",\"SelectionMode\":\"EnterPath\"},\"Private.BrowserFolderName\":\"Inbox\",\"Public.MaxResults\":\"50\",\"Public.Connector\":\"uipath-microsoft-outlook365\",\"Private.DisplayName\":\"Get Email List\"}", - "Status": 2, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "SpanType": "Activity", - "JobKey": "b7974725-4438-4315-a9cc-9ec41f8acd62" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/README.md deleted file mode 100644 index b31f31d2d2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# O365 Interactive Token Timeout — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The Microsoft 365 Scope in the **ERN** process is configured for **Interactive Token** authentication (no account bound — the Account field still reads "Please select an account.", Integration Service off), which is the wrong and fragile auth choice for this Agent/StudioPro execution context. Interactive Token requires a human to complete a browser sign-in within a 30-second window; in this no-bound-account agent run nobody completed it, so authentication timed out and the job faulted. Switch the scope to app-only authentication (Application ID + Client Secret, or Application ID + Certificate). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name o365-interactive-token-timeout --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/a7893322be.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/a7893322be.json deleted file mode 100644 index f00a9e7f3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/a7893322be.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "State": "Faulted", - "Source": "Agent", - "SourceType": "Agent", - "Info": "Get Email List: Authentication failed while resolving mail folder 'Inbox'.\n\nSystem.TimeoutException: The client did not complete the authentication after 30 seconds, and as a result the operation was canceled. Authentication type: InteractiveToken.\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.LoginWithInteractiveTokenAsync()\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.ResolveAccessTokenAsync()\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.AuthenticateAsync()\n at Microsoft.Graph.AuthenticationHandler.SendAsync()\n at Microsoft.Graph.UserMailFoldersCollectionRequest.GetAsync()\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailFolderByPathAsync()\n at UiPath.MicrosoftOffice365.Activities.Mail.BaseMailFolderArgument.ResolveAsync()\n at UiPath.MicrosoftOffice365.Activities.Mail.GetEmailListConnections.SafeExecuteAsync()\n at UiPath.Shared.Activities.Utilities.TaskExtensions.TimeoutAfter()", - "ReleaseName": "ERN", - "Type": "Attended", - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "EntryPointPath": "Main.xaml", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ErrorCode": "Robot", - "TraceId": "b797472544384315a9cc9ec41f8acd62" - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/manifest.json deleted file mode 100644 index d3a4bb699d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/m/r/manifest.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py; docsai rules switched to passthrough per tests/tasks/uipath-troubleshoot/CLAUDE.md (query strings vary per run).", - "rules": [ - { - "match": "or folders list --output json", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --output json", - "file": "15736f592f.json" - }, - { - "match": "or jobs get b7974725-4438-4315-a9cc-9ec41f8acd62 --output json", - "file": "a7893322be.json" - }, - { - "match": "or jobs traces b7974725-4438-4315-a9cc-9ec41f8acd62 --output json", - "file": "99d0038951.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by rules returns an empty array." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/uip-fixture.json new file mode 100644 index 0000000000..5501205712 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 21,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b7974725-4438-4315-a9cc-9ec41f8acd62", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"Info\": \"Get Email List: Authentication failed while resolving mail folder 'Inbox'.\\n\\nSystem.TimeoutException: The client did not complete the authentication after 30 seconds, and as a result the operation was canceled. Authentication type: InteractiveToken.\\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.LoginWithInteractiveTokenAsync()\\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.ResolveAccessTokenAsync()\\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.AuthenticateAsync()\\n at Microsoft.Graph.AuthenticationHandler.SendAsync()\\n at Microsoft.Graph.UserMailFoldersCollectionRequest.GetAsync()\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetMailFolderByPathAsync()\\n at UiPath.MicrosoftOffice365.Activities.Mail.BaseMailFolderArgument.ResolveAsync()\\n at UiPath.MicrosoftOffice365.Activities.Mail.GetEmailListConnections.SafeExecuteAsync()\\n at UiPath.Shared.Activities.Utilities.TaskExtensions.TimeoutAfter()\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b797472544384315a9cc9ec41f8acd62\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "b7974725-4438-4315-a9cc-9ec41f8acd62", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"00000000-0000-0000-0000-000000000001\",\r\n \"TraceId\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.TimeoutException: The client did not complete the authentication after 30 seconds, and as a result the operation was canceled. Authentication type: InteractiveToken.\\\\n at UiPath.MicrosoftOffice365.Services.GraphClientServiceFactory.LoginWithInteractiveTokenAsync()\\\\n at UiPath.Shared.Activities.Utilities.TaskExtensions.TimeoutAfter()\\\"}\",\r\n \"Status\": 2,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"SpanType\": \"RobotJob\",\r\n \"JobKey\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\"\r\n },\r\n {\r\n \"Id\": \"00000000-0000-0000-0000-000000000002\",\r\n \"TraceId\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"ParentId\": \"00000000-0000-0000-0000-000000000001\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-09T10:49:39.313Z\",\r\n \"EndTime\": \"2026-06-09T10:49:39.351Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"SpanType\": \"Activity\",\r\n \"JobKey\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\"\r\n },\r\n {\r\n \"Id\": \"00000000-0000-0000-0000-000000000003\",\r\n \"TraceId\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"ParentId\": \"00000000-0000-0000-0000-000000000001\",\r\n \"Name\": \"Get Email List\",\r\n \"StartTime\": \"2026-06-09T10:49:39.921Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.TimeoutException: The client did not complete the authentication after 30 seconds, and as a result the operation was canceled. Authentication type: InteractiveToken.\\\",\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.GetEmailListConnections\\\",\\\"Private.MailFolderArg\\\":{\\\"FolderName\\\":\\\"Inbox\\\",\\\"SelectionMode\\\":\\\"EnterPath\\\"},\\\"Private.BrowserFolderName\\\":\\\"Inbox\\\",\\\"Public.MaxResults\\\":\\\"50\\\",\\\"Public.Connector\\\":\\\"uipath-microsoft-outlook365\\\",\\\"Private.DisplayName\\\":\\\"Get Email List\\\"}\",\r\n \"Status\": 2,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"SpanType\": \"Activity\",\r\n \"JobKey\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/task.yaml index b9117f844f..889ef199db 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-interactive-token-timeout/task.yaml @@ -17,21 +17,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/README.md new file mode 100644 index 0000000000..618e065c06 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/README.md @@ -0,0 +1,59 @@ +# O365 Send Mail Missing Attachment — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `NotificationMailer` (key `74db8e32-8535-40b2-9b40-62540456adab`), +faulted with a raw `System.IO.FileNotFoundException: File does not exist: +C:\Temp\missing-attachment-repro.pdf` thrown by the legacy O365 `SendMail` +activity. The package's `AttachmentsHelpers.EnsureFileExists` check threw +while assembling attachments — before any Microsoft Graph send call. Trace +spans show the Microsoft 365 Scope opened cleanly (auth OK), and no +`ErrorInvalidRecipients` / `ErrorSendAsDenied` / `ErrorMessageSizeExceeded` +codes appear anywhere — the other send-rejection causes are ruled out. Fix: +stage the file on the runtime machine and/or correct the `Attachments` path. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info/logs/traces carry the full fault stack). + +### Local filesystem check (not mocked) + +The original investigation also verified the missing path with a LOCAL +PowerShell `Test-Path` on the runtime machine (the local hostname matched the +job's host machine). That is not a `uip` call, so it is intentionally NOT in +the mock manifest. The eval agent may attempt the same check inside the +sandbox — `C:\Temp\missing-attachment-repro.pdf` is absent there too, which is +consistent with the recorded evidence. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs (the generator's transcript pass is unreliable for this session). If +the source investigation changes, update the fixture JSONs directly and re-run +the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-sendmail-missing-attachment --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/0c9cb3d612.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/0c9cb3d612.json deleted file mode 100644 index a4a90c793d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/0c9cb3d612.json +++ /dev/null @@ -1,376 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 26, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/4d787a213d.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/4d787a213d.json deleted file mode 100644 index db97450554..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/4d787a213d.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T18:15:38.260Z", - "JobId": 4015110 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T18:15:34.937Z", - "JobId": 4015110 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T18:15:34.357Z", - "JobId": 4015110 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/4f52853c5d.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/4f52853c5d.json deleted file mode 100644 index 0b80a495e2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/4f52853c5d.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ca8c5638-937d-44f8-a56a-e47ea122ddfd", - "Info": "File does not exist: C:\\Temp\\missing-attachment-repro.pdf\r\n\r\nin Main.xaml\r\n at SendMail \"Send Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Send Mail rejected repro (missing attachment)\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileNotFoundException: File does not exist: C:\\Temp\\missing-attachment-repro.pdf\r\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\r\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T18:15:34.323Z", - "ReleaseName": "NotificationMailer", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53469, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T18:15:38.250Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "74db8e32853540b29b4062540456adab", - "Id": 4015110 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/9daf4a08fe.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/9daf4a08fe.json deleted file mode 100644 index 2997635420..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/9daf4a08fe.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-61b5-1c570c22d5c4", - "TraceId": "74db8e32-8535-40b2-9b40-62540456adab", - "ParentId": null, - "Name": "Robot Job 74db8e32-8535-40b2-9b40-62540456adab", - "StartTime": "2026-06-10T18:15:35.3332897Z", - "EndTime": "2026-06-10T18:15:37.5528577Z", - "Attributes": "{\"Exception\":\"System.IO.FileNotFoundException: File does not exist: C:\\\\Temp\\\\missing-attachment-repro.pdf\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "ded34256-4a90-41cc-a41c-8fab58e73c65", - "JobKey": "74db8e32-8535-40b2-9b40-62540456adab", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T18:15:38.1040882Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-a176-c9d347734272", - "TraceId": "74db8e32-8535-40b2-9b40-62540456adab", - "ParentId": "00000000-0000-0000-61b5-1c570c22d5c4", - "Name": "Microsoft 365 Scope", - "StartTime": "2026-06-10T18:15:35.4309749Z", - "EndTime": "2026-06-10T18:15:35.4677404Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\",\"Public.ConfigLocation\":\"PropertiesPanel\",\"Public.RuntimeItemInputMode\":\"Browse\",\"Public.AuthenticationType\":\"InteractiveToken\",\"Public.Environment\":\"Global\",\"Public.OAuthApplication\":\"UIPATH\",\"Public.DataStoreLocation\":\"DISK\",\"Public.UseConnectionService\":\"False\",\"Private.Account\":\"Please select an account.\",\"Private.DisplayName\":\"Microsoft 365 Scope\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "ded34256-4a90-41cc-a41c-8fab58e73c65", - "JobKey": "74db8e32-8535-40b2-9b40-62540456adab", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T18:15:38.1040882Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-3455-a97c4bda37d6", - "TraceId": "74db8e32-8535-40b2-9b40-62540456adab", - "ParentId": "00000000-0000-0000-61b5-1c570c22d5c4", - "Name": "Send Mail", - "StartTime": "2026-06-10T18:15:37.4971521Z", - "EndTime": "2026-06-10T18:15:37.4988382Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.SendMail\",\"Private.Body\":\"Repro: configured attachment path does not exist on the runtime machine.\",\"Private.Subject\":\"Send-rejected repro\",\"Public.IsDraft\":\"False\",\"Private.Attachments\":{\"0\":\"@\\\"C:\\\\Temp\\\\missing-attachment-repro.pdf\\\"\"},\"Public.IsBodyHTML\":\"False\",\"Private.To\":\"new string[] { \\\"recipient@example.com\\\" }\",\"Public.Importance\":\"Normal\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"Send Mail\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "ded34256-4a90-41cc-a41c-8fab58e73c65", - "JobKey": "74db8e32-8535-40b2-9b40-62540456adab", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T18:15:38.1040882Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/README.md deleted file mode 100644 index c79b5ea777..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# O365 Send Mail Missing Attachment — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `NotificationMailer` (key `74db8e32-8535-40b2-9b40-62540456adab`), -faulted with a raw `System.IO.FileNotFoundException: File does not exist: -C:\Temp\missing-attachment-repro.pdf` thrown by the legacy O365 `SendMail` -activity. The package's `AttachmentsHelpers.EnsureFileExists` check threw -while assembling attachments — before any Microsoft Graph send call. Trace -spans show the Microsoft 365 Scope opened cleanly (auth OK), and no -`ErrorInvalidRecipients` / `ErrorSendAsDenied` / `ErrorMessageSizeExceeded` -codes appear anywhere — the other send-rejection causes are ruled out. Fix: -stage the file on the runtime machine and/or correct the `Attachments` path. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info/logs/traces carry the full fault stack). - -### Local filesystem check (not mocked) - -The original investigation also verified the missing path with a LOCAL -PowerShell `Test-Path` on the runtime machine (the local hostname matched the -job's host machine). That is not a `uip` call, so it is intentionally NOT in -the mock manifest. The eval agent may attempt the same check inside the -sandbox — `C:\Temp\missing-attachment-repro.pdf` is absent there too, which is -consistent with the recorded evidence. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs (the generator's transcript pass is unreliable for this session). If -the source investigation changes, update the fixture JSONs directly and re-run -the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-sendmail-missing-attachment --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/a086140bde.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/a086140bde.json deleted file mode 100644 index 90b1244475..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/a086140bde.json +++ /dev/null @@ -1,250 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 17, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/c6b496afe9.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/c6b496afe9.json deleted file mode 100644 index ad9f687b84..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/c6b496afe9.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T18:15:37.524Z", - "Message": "Send Mail: File does not exist: C:\\Temp\\missing-attachment-repro.pdf\r\nin Main.xaml\r\n at SendMail \"Send Mail\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Send Mail rejected repro (missing attachment)\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IO.FileNotFoundException\r\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\r\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/manifest.json deleted file mode 100644 index d789f03d3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/m/r/manifest.json +++ /dev/null @@ -1,90 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 74db8e32-8535-40b2-9b40-62540456adab", - "file": "4f52853c5d.json" - }, - { - "match": "or jobs logs 74db8e32-8535-40b2-9b40-62540456adab --level Error", - "file": "c6b496afe9.json" - }, - { - "match": "or jobs logs 74db8e32-8535-40b2-9b40-62540456adab", - "file": "c6b496afe9.json" - }, - { - "match": "or jobs history 74db8e32-8535-40b2-9b40-62540456adab", - "file": "4d787a213d.json" - }, - { - "match": "or jobs traces 74db8e32-8535-40b2-9b40-62540456adab", - "file": "9daf4a08fe.json" - }, - { - "match": "traces spans get --job-key 74db8e32-8535-40b2-9b40-62540456adab", - "file": "9daf4a08fe.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same span payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default and the agent loses the scope-opened-cleanly / attachment-property evidence." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "a086140bde.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "0c9cb3d612.json" - }, - { - "match": "or jobs list", - "file": "0c9cb3d612.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 74db8e32-8535-40b2-9b40-62540456adab", - "min": 1, - "description": "Agent must fetch the failing job to read the raw System.IO.FileNotFoundException (File does not exist: C:\\Temp\\missing-attachment-repro.pdf) from AttachmentsHelpers.EnsureFileExists." - }, - { - "pattern": "traces", - "min": 0, - "description": "OPTIONAL — spans corroborate: Microsoft 365 Scope completed cleanly (auth OK) and the SendMail span's Private.Attachments carries the missing path. Root cause is already fully visible in jobs get." - }, - { - "pattern": "or jobs logs 74db8e32-8535-40b2-9b40-62540456adab", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history 74db8e32-8535-40b2-9b40-62540456adab", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/uip-fixture.json new file mode 100644 index 0000000000..c1da893950 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/data/uip-fixture.json @@ -0,0 +1,156 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 17,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "74db8e32-8535-40b2-9b40-62540456adab", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T18:15:37.524Z\",\r\n \"Message\": \"Send Mail: File does not exist: C:\\\\Temp\\\\missing-attachment-repro.pdf\\r\\nin Main.xaml\\r\\n at SendMail \\\"Send Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Send Mail rejected repro (missing attachment)\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "74db8e32-8535-40b2-9b40-62540456adab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-61b5-1c570c22d5c4\",\r\n \"TraceId\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"StartTime\": \"2026-06-10T18:15:35.3332897Z\",\r\n \"EndTime\": \"2026-06-10T18:15:37.5528577Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.IO.FileNotFoundException: File does not exist: C:\\\\\\\\Temp\\\\\\\\missing-attachment-repro.pdf\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"ded34256-4a90-41cc-a41c-8fab58e73c65\",\r\n \"JobKey\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T18:15:38.1040882Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-a176-c9d347734272\",\r\n \"TraceId\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ParentId\": \"00000000-0000-0000-61b5-1c570c22d5c4\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T18:15:35.4309749Z\",\r\n \"EndTime\": \"2026-06-10T18:15:35.4677404Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ded34256-4a90-41cc-a41c-8fab58e73c65\",\r\n \"JobKey\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T18:15:38.1040882Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-3455-a97c4bda37d6\",\r\n \"TraceId\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ParentId\": \"00000000-0000-0000-61b5-1c570c22d5c4\",\r\n \"Name\": \"Send Mail\",\r\n \"StartTime\": \"2026-06-10T18:15:37.4971521Z\",\r\n \"EndTime\": \"2026-06-10T18:15:37.4988382Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.SendMail\\\",\\\"Private.Body\\\":\\\"Repro: configured attachment path does not exist on the runtime machine.\\\",\\\"Private.Subject\\\":\\\"Send-rejected repro\\\",\\\"Public.IsDraft\\\":\\\"False\\\",\\\"Private.Attachments\\\":{\\\"0\\\":\\\"@\\\\\\\"C:\\\\\\\\Temp\\\\\\\\missing-attachment-repro.pdf\\\\\\\"\\\"},\\\"Public.IsBodyHTML\\\":\\\"False\\\",\\\"Private.To\\\":\\\"new string[] { \\\\\\\"recipient@example.com\\\\\\\" }\\\",\\\"Public.Importance\\\":\\\"Normal\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Send Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ded34256-4a90-41cc-a41c-8fab58e73c65\",\r\n \"JobKey\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T18:15:38.1040882Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 26,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "74db8e32-8535-40b2-9b40-62540456adab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ca8c5638-937d-44f8-a56a-e47ea122ddfd\",\r\n \"Info\": \"File does not exist: C:\\\\Temp\\\\missing-attachment-repro.pdf\\r\\n\\r\\nin Main.xaml\\r\\n at SendMail \\\"Send Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Send Mail rejected repro (missing attachment)\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: File does not exist: C:\\\\Temp\\\\missing-attachment-repro.pdf\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\",\r\n \"ReleaseName\": \"NotificationMailer\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53469,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"74db8e32853540b29b4062540456adab\",\r\n \"Id\": 4015110\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "74db8e32-8535-40b2-9b40-62540456adab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T18:15:37.524Z\",\r\n \"Message\": \"Send Mail: File does not exist: C:\\\\Temp\\\\missing-attachment-repro.pdf\\r\\nin Main.xaml\\r\\n at SendMail \\\"Send Mail\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Send Mail rejected repro (missing attachment)\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\\r\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "74db8e32-8535-40b2-9b40-62540456adab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T18:15:38.260Z\",\r\n \"JobId\": 4015110\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T18:15:34.937Z\",\r\n \"JobId\": 4015110\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T18:15:34.357Z\",\r\n \"JobId\": 4015110\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "74db8e32-8535-40b2-9b40-62540456adab" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-61b5-1c570c22d5c4\",\r\n \"TraceId\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"StartTime\": \"2026-06-10T18:15:35.3332897Z\",\r\n \"EndTime\": \"2026-06-10T18:15:37.5528577Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.IO.FileNotFoundException: File does not exist: C:\\\\\\\\Temp\\\\\\\\missing-attachment-repro.pdf\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.EnsureFileExists(String path)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.AttachmentsHelpers.GetAttachments(AsyncCodeActivityContext context, IEnumerable`1 attachmentsCollection, List`1 attachments)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.SendMail.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"ded34256-4a90-41cc-a41c-8fab58e73c65\",\r\n \"JobKey\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T18:15:38.1040882Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-a176-c9d347734272\",\r\n \"TraceId\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ParentId\": \"00000000-0000-0000-61b5-1c570c22d5c4\",\r\n \"Name\": \"Microsoft 365 Scope\",\r\n \"StartTime\": \"2026-06-10T18:15:35.4309749Z\",\r\n \"EndTime\": \"2026-06-10T18:15:35.4677404Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Office365ApplicationScope\\\",\\\"Public.ConfigLocation\\\":\\\"PropertiesPanel\\\",\\\"Public.RuntimeItemInputMode\\\":\\\"Browse\\\",\\\"Public.AuthenticationType\\\":\\\"InteractiveToken\\\",\\\"Public.Environment\\\":\\\"Global\\\",\\\"Public.OAuthApplication\\\":\\\"UIPATH\\\",\\\"Public.DataStoreLocation\\\":\\\"DISK\\\",\\\"Public.UseConnectionService\\\":\\\"False\\\",\\\"Private.Account\\\":\\\"Please select an account.\\\",\\\"Private.DisplayName\\\":\\\"Microsoft 365 Scope\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ded34256-4a90-41cc-a41c-8fab58e73c65\",\r\n \"JobKey\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T18:15:38.1040882Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-3455-a97c4bda37d6\",\r\n \"TraceId\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ParentId\": \"00000000-0000-0000-61b5-1c570c22d5c4\",\r\n \"Name\": \"Send Mail\",\r\n \"StartTime\": \"2026-06-10T18:15:37.4971521Z\",\r\n \"EndTime\": \"2026-06-10T18:15:37.4988382Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.SendMail\\\",\\\"Private.Body\\\":\\\"Repro: configured attachment path does not exist on the runtime machine.\\\",\\\"Private.Subject\\\":\\\"Send-rejected repro\\\",\\\"Public.IsDraft\\\":\\\"False\\\",\\\"Private.Attachments\\\":{\\\"0\\\":\\\"@\\\\\\\"C:\\\\\\\\Temp\\\\\\\\missing-attachment-repro.pdf\\\\\\\"\\\"},\\\"Public.IsBodyHTML\\\":\\\"False\\\",\\\"Private.To\\\":\\\"new string[] { \\\\\\\"recipient@example.com\\\\\\\" }\\\",\\\"Public.Importance\\\":\\\"Normal\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"Send Mail\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"ded34256-4a90-41cc-a41c-8fab58e73c65\",\r\n \"JobKey\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T18:15:38.1040882Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 26,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 74db8e32-8535-40b2-9b40-62540456adab", + "min": 1, + "description": "Agent must fetch the failing job to read the raw System.IO.FileNotFoundException (File does not exist: C:\\Temp\\missing-attachment-repro.pdf) from AttachmentsHelpers.EnsureFileExists." + }, + { + "pattern": "traces", + "min": 0, + "description": "OPTIONAL — spans corroborate: Microsoft 365 Scope completed cleanly (auth OK) and the SendMail span's Private.Attachments carries the missing path. Root cause is already fully visible in jobs get." + }, + { + "pattern": "or jobs logs 74db8e32-8535-40b2-9b40-62540456adab", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history 74db8e32-8535-40b2-9b40-62540456adab", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/task.yaml index f6669fc785..a4002a1c79 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sendmail-missing-attachment/task.yaml @@ -20,21 +20,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/README.md new file mode 100644 index 0000000000..2487d9617a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/README.md @@ -0,0 +1,36 @@ +# O365 Sharing Link Access Denied — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The OneDrive/SharePoint sharing link that the Read Range activity uses to locate its Excel workbook is no longer valid for the job's Microsoft 365 connection — the link was revoked, its sharing scope was changed, or the underlying item's access was removed for the connection's account. When Read Range tried to resolve that sharing link through Microsoft Graph, Graph returned `accessDenied` (HTTP 403), faulting the job before the range was ever read. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name o365-sharing-link-access-denied --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/6dfb4c4383.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/6dfb4c4383.json deleted file mode 100644 index fb3875e214..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/6dfb4c4383.json +++ /dev/null @@ -1,208 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 14, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/README.md deleted file mode 100644 index 2d456869d1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# O365 Sharing Link Access Denied — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The OneDrive/SharePoint sharing link that the Read Range activity uses to locate its Excel workbook is no longer valid for the job's Microsoft 365 connection — the link was revoked, its sharing scope was changed, or the underlying item's access was removed for the connection's account. When Read Range tried to resolve that sharing link through Microsoft Graph, Graph returned `accessDenied` (HTTP 403), faulting the job before the range was ever read. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name o365-sharing-link-access-denied --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/d697dbff7c.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/d697dbff7c.json deleted file mode 100644 index 4e0167b1f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/d697dbff7c.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "Info": "The caller doesn't have permission to perform the action.\r\n\r\nin Main.xaml\r\n at ReadRangeConnections`1 \"Read Range\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Transient service error / timeout (5xx) repro\"\r\n at Main \"Main\"\r\n\r\nUiPath.MicrosoftOffice365.Office365Exception: The caller doesn't have permission to perform the action.\r\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Code: accessDenied\r\nMessage: The sharing link no longer exists, or you do not have permission to access it.\r\nClientRequestId: a5ebb0db-84ab-470e-a4ac-2b1c05cc233e\r\nAdditionalData:\r\n\tonedrive.linkFeatures@odata.type: #Collection(oneDrive.linkFeatures)\r\n\t@onedrive.linkFeatures: []\r\n\r\n ---> Microsoft.Graph.ServiceException: Code: accessDenied\r\nMessage: The sharing link no longer exists, or you do not have permission to access it.\r\nClientRequestId: a5ebb0db-84ab-470e-a4ac-2b1c05cc233e\r\nAdditionalData:\r\n\tonedrive.linkFeatures@odata.type: #Collection(oneDrive.linkFeatures)\r\n\t@onedrive.linkFeatures: []\r\n\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.DriveItemRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetItemAsync(GetItemRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.FromRequest(BaseRangeRequest request, Boolean hasHeaders, Boolean ignoreEmptyValues, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.ReadRangeAsync[T](ReadRangeRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Excel.ReadRangeConnections`1.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.DriveItemRequest.GetAsync(CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetItemAsync(GetItemRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.FromRequest(BaseRangeRequest request, Boolean hasHeaders, Boolean ignoreEmptyValues, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.ReadRangeAsync[T](ReadRangeRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Excel.ReadRangeConnections`1.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-09T12:52:19.450Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53466, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-09T12:52:22.867Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "de11f5f7a48a4f76bb30ada33ea307fe", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 4015054 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/dabecc7dbd.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/dabecc7dbd.json deleted file mode 100644 index 25c7ea5f37..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/dabecc7dbd.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "HTTP 503: no healthy upstream", - "Instructions": "Verify the connector key and folder key, then retry. Use 'uip is connections list --help' for accepted options.\n\nDetails: no healthy upstream" -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/manifest.json deleted file mode 100644 index bb5c26c2bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/m/r/manifest.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py. Adjusted: docsai -> passthrough (per tests CLAUDE.md); the original session used a $K shell variable for the job key AND piped each call to a file with exit-marker echoes, so the recorded 'or jobs get $K' rule was rewritten to the literal key de11f5f7-... and the corrupted exit-marker fixtures (EXIT=0 trailers) were rebuilt from the investigation raw/ responses with PII scrubbed. Match strings shortened to robust substrings for the substring dispatcher.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "6dfb4c4383.json" - }, - { - "match": "or jobs get de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "file": "d697dbff7c.json" - }, - { - "match": "is connections list", - "file": "dabecc7dbd.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by rules returns an empty array." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/uip-fixture.json new file mode 100644 index 0000000000..215774f214 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/data/uip-fixture.json @@ -0,0 +1,52 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 14,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "de11f5f7-a48a-4f76-bb30-ada33ea307fe" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"Info\": \"The caller doesn't have permission to perform the action.\\r\\n\\r\\nin Main.xaml\\r\\n at ReadRangeConnections`1 \\\"Read Range\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Transient service error / timeout (5xx) repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.MicrosoftOffice365.Office365Exception: The caller doesn't have permission to perform the action.\\r\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Code: accessDenied\\r\\nMessage: The sharing link no longer exists, or you do not have permission to access it.\\r\\nClientRequestId: a5ebb0db-84ab-470e-a4ac-2b1c05cc233e\\r\\nAdditionalData:\\r\\n\\tonedrive.linkFeatures@odata.type: #Collection(oneDrive.linkFeatures)\\r\\n\\t@onedrive.linkFeatures: []\\r\\n\\r\\n ---> Microsoft.Graph.ServiceException: Code: accessDenied\\r\\nMessage: The sharing link no longer exists, or you do not have permission to access it.\\r\\nClientRequestId: a5ebb0db-84ab-470e-a4ac-2b1c05cc233e\\r\\nAdditionalData:\\r\\n\\tonedrive.linkFeatures@odata.type: #Collection(oneDrive.linkFeatures)\\r\\n\\t@onedrive.linkFeatures: []\\r\\n\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.DriveItemRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetItemAsync(GetItemRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.FromRequest(BaseRangeRequest request, Boolean hasHeaders, Boolean ignoreEmptyValues, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.ReadRangeAsync[T](ReadRangeRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Excel.ReadRangeConnections`1.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.BaseRequest.SendRequestAsync(Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.BaseRequest.SendAsync[T](Object serializableObject, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.DriveItemRequest.GetAsync(CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetItemAsync(GetItemRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.FromRequest(BaseRangeRequest request, Boolean hasHeaders, Boolean ignoreEmptyValues, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.ReadRangeAsync[T](ReadRangeRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Excel.ReadRangeConnections`1.SafeExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53466,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"de11f5f7a48a4f76bb30ada33ea307fe\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 4015054\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"HTTP 503: no healthy upstream\",\r\n \"Instructions\": \"Verify the connector key and folder key, then retry. Use 'uip is connections list --help' for accepted options.\\n\\nDetails: no healthy upstream\"\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/task.yaml index 0f8820f6d7..c137809689 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-sharing-link-access-denied/task.yaml @@ -18,21 +18,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/README.md new file mode 100644 index 0000000000..342dcc2bb7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/README.md @@ -0,0 +1,82 @@ +# O365 Email Trigger — Integration Service 503 Outage — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `MailEventListener` (key `1db8bac1-ecc2-4670-843e-ff5f9a26c532`), +faulted after ~2.4 s when its **New Email Received** trigger activity +(`NewEmailReceived`, "New Email Received via dead connection (expect +ConnectionHttpException)" in `Main.xaml`) could not reach +the Integration Service connection layer. The HTTP request received 503; the +connection client exhausted its internal retries +(`HttpConnectionServiceClient.RequestWithRetryAsync`) and threw +`UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service +Unavailable`, surfaced as `Office365Exception: Service Unavailable`. No +Microsoft Graph call was reached — the failure is in the UiPath Integration +Service layer, not Outlook. Fix: retry once the connection service recovers +(verify via an IS connections probe) and escalate as a service incident if +the 503 persists. + +### Live-probe evidence pattern + +This scenario's distinguishing evidence is a **live probe of the platform**, +not just job artifacts: the original investigation ran `uip is connections +list --folder-key ` three times (~35 min after the job) and received +HTTP 503 "no healthy upstream" on every attempt — confirming a real, +ongoing Integration Service outage. The 5xx status class rules out the +sibling causes from the playbook's branch table: connection deleted/not +shared (404-class) and authorization expired/revoked (401/403-class). The +mock replays the recorded 503 Failure payload (exit code 1) for every +`is connections list` variant so the eval agent can reproduce the probe. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info/logs/traces carry the full +ConnectionHttpException stack; the live IS probe carries the outage proof). + +### Trace spans + +Three spans were recorded for this job and both trace retrieval forms +(`or jobs traces ` and `traces spans get --job-key `) replay them: +the RobotJob span (full exception stack in Attributes), the +`NewEmailReceived` activity span (connector `uipath-microsoft-outlook365`, +`UseConnectionService=True`), and the "operation failed" error span. + +### IS 503 fixture choice + +The three recorded probe attempts (`triage-is-connections.json`, +`triage-is-connections-retry.json`, `H1-is-connections-attempt3.json`) +were byte-identical, so one representative fixture +(`is-connections-list-folder-key-1965a46b-db4e-469e-aaaa-7e0b379cb34d-503.json`) +answers all probe retries. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs. If the source investigation changes, update the fixture JSONs +directly and re-run the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-trigger-connection-503 --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/16d8e9e6ea.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/16d8e9e6ea.json deleted file mode 100644 index 981a4799a8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/16d8e9e6ea.json +++ /dev/null @@ -1,432 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 30, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/437e071e60.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/437e071e60.json deleted file mode 100644 index 6c76878d0b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/437e071e60.json +++ /dev/null @@ -1,306 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 21, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/7762847a0f.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/7762847a0f.json deleted file mode 100644 index f700b5d907..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/7762847a0f.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T19:32:28.579Z", - "Message": "New Email Received: Service Unavailable\r\nin Main.xaml\r\n at NewEmailReceived \"New Email Received\"\r\n at Sequence \"O365 email trigger connection failure repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.MicrosoftOffice365.Office365Exception\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/9b764201dd.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/9b764201dd.json deleted file mode 100644 index 53c5aa998e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/9b764201dd.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-10T19:32:29.087Z", - "JobId": 4015114 - }, - { - "State": "Running", - "Timestamp": "2026-06-10T19:32:26.620Z", - "JobId": 4015114 - }, - { - "State": "Pending", - "Timestamp": "2026-06-10T19:32:26.277Z", - "JobId": 4015114 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/README.md deleted file mode 100644 index ddbfa9dda4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/README.md +++ /dev/null @@ -1,81 +0,0 @@ -# O365 Email Trigger — Integration Service 503 Outage — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `MailEventListener` (key `1db8bac1-ecc2-4670-843e-ff5f9a26c532`), -faulted after ~2.4 s when its **New Email Received** trigger activity -(`NewEmailReceived`, "New Email Received via dead connection (expect -ConnectionHttpException)" in `Main.xaml`) could not reach -the Integration Service connection layer. The HTTP request received 503; the -connection client exhausted its internal retries -(`HttpConnectionServiceClient.RequestWithRetryAsync`) and threw -`UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service -Unavailable`, surfaced as `Office365Exception: Service Unavailable`. No -Microsoft Graph call was reached — the failure is in the UiPath Integration -Service layer, not Outlook. Fix: retry once the connection service recovers -(verify via an IS connections probe) and escalate as a service incident if -the 503 persists. - -### Live-probe evidence pattern - -This scenario's distinguishing evidence is a **live probe of the platform**, -not just job artifacts: the original investigation ran `uip is connections -list --folder-key ` three times (~35 min after the job) and received -HTTP 503 "no healthy upstream" on every attempt — confirming a real, -ongoing Integration Service outage. The 5xx status class rules out the -sibling causes from the playbook's branch table: connection deleted/not -shared (404-class) and authorization expired/revoked (401/403-class). The -mock replays the recorded 503 Failure payload (exit code 1) for every -`is connections list` variant so the eval agent can reproduce the probe. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info/logs/traces carry the full -ConnectionHttpException stack; the live IS probe carries the outage proof). - -### Trace spans - -Three spans were recorded for this job and both trace retrieval forms -(`or jobs traces ` and `traces spans get --job-key `) replay them: -the RobotJob span (full exception stack in Attributes), the -`NewEmailReceived` activity span (connector `uipath-microsoft-outlook365`, -`UseConnectionService=True`), and the "operation failed" error span. - -### IS 503 fixture choice - -The three recorded probe attempts (`triage-is-connections.json`, -`triage-is-connections-retry.json`, `H1-is-connections-attempt3.json`) -were byte-identical, so one representative fixture -(`is-connections-list-folder-key-1965a46b-db4e-469e-aaaa-7e0b379cb34d-503.json`) -answers all probe retries. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs. If the source investigation changes, update the fixture JSONs -directly and re-run the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-trigger-connection-503 --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/b3f1385c00.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/b3f1385c00.json deleted file mode 100644 index 06a3e3df27..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/b3f1385c00.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "01a8cd00-0985-434a-8ffd-3a81a9fbad66", - "Info": "Service Unavailable\r\n\r\nin Main.xaml\r\n at NewEmailReceived \"New Email Received\"\r\n at Sequence \"O365 email trigger connection failure repro\"\r\n at Main \"Main\"\r\n\r\nUiPath.MicrosoftOffice365.Office365Exception: Service Unavailable\r\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Service Unavailable\r\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service Unavailable\r\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\r\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\r\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\r\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\r\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-10T19:32:26.273Z", - "ReleaseName": "MailEventListener", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53473, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T19:32:29.057Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "1db8bac1ecc24670843eff5f9a26c532", - "Id": 4015114 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/d4abf1a83b.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/d4abf1a83b.json deleted file mode 100644 index 6b8a0e2edd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/d4abf1a83b.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-629c-5e83d4c6a10f", - "TraceId": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "ParentId": null, - "Name": "Robot Job 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "StartTime": "2026-06-10T19:32:26.9708545Z", - "EndTime": "2026-06-10T19:32:28.6336416Z", - "Attributes": "{\"Exception\":\"UiPath.MicrosoftOffice365.Office365Exception: Service Unavailable\\r\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Service Unavailable\\r\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service Unavailable\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\r\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "a44a2ebf-2f3e-49f7-ad56-2601f66b3c65", - "JobKey": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:32:28.9110662Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-c852-4be99f090254", - "TraceId": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "ParentId": "00000000-0000-0000-629c-5e83d4c6a10f", - "Name": "New Email Received", - "StartTime": "2026-06-10T19:32:27.0496037Z", - "EndTime": "2026-06-10T19:32:27.0946303Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived\",\"Private.MailFolderArg\":{\"FolderName\":\"Inbox\",\"SelectionMode\":\"EnterPath\"},\"Private.BrowserFolderName\":\"Inbox\",\"Public.MarkAsRead\":\"False\",\"Private.Result\":\"email\",\"Public.Connector\":\"uipath-microsoft-outlook365\",\"Public.WithAttachmentsOnly\":\"False\",\"Public.IncludeAttachments\":\"False\",\"Public.UseSharedMailbox\":\"False\",\"Private.MailboxArg\":{\"UseSharedMailbox\":\"False\"},\"Public.ConnectionId\":\"00000000-0000-0000-0000-000000000000\",\"Public.UseConnectionService\":\"True\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"New Email Received\"}", - "Status": 1, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "a44a2ebf-2f3e-49f7-ad56-2601f66b3c65", - "JobKey": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:32:28.0806145Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - }, - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-e51f-b0044f677568", - "TraceId": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "ParentId": "00000000-0000-0000-c852-4be99f090254", - "Name": "The operation failed with exception: 'Service Unavailable'", - "StartTime": "2026-06-10T19:32:28.5632205Z", - "EndTime": "2026-06-10T19:32:28.5637664Z", - "Attributes": "{\"Private.Last requests\":\"\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "a44a2ebf-2f3e-49f7-ad56-2601f66b3c65", - "JobKey": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-10T19:32:28.9110662Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/dabecc7dbd.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/dabecc7dbd.json deleted file mode 100644 index 25c7ea5f37..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/dabecc7dbd.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "HTTP 503: no healthy upstream", - "Instructions": "Verify the connector key and folder key, then retry. Use 'uip is connections list --help' for accepted options.\n\nDetails: no healthy upstream" -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/ef63e67445.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/ef63e67445.json deleted file mode 100644 index aa3ae34f2c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/ef63e67445.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-10T19:32:28.943Z", - "Message": "MailEventListener execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T19:32:28.579Z", - "Message": "New Email Received: Service Unavailable\r\nin Main.xaml\r\n at NewEmailReceived \"New Email Received\"\r\n at Sequence \"O365 email trigger connection failure repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.MicrosoftOffice365.Office365Exception\r\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-10T19:32:26.995Z", - "Message": "MailEventListener execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/manifest.json deleted file mode 100644 index 3722c02a26..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/m/r/manifest.json +++ /dev/null @@ -1,104 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "file": "b3f1385c00.json" - }, - { - "match": "or jobs logs 1db8bac1-ecc2-4670-843e-ff5f9a26c532 --level Error", - "file": "7762847a0f.json" - }, - { - "match": "or jobs logs 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "file": "ef63e67445.json", - "_doc": "Full log set (execution started -> Error -> execution ended) for unfiltered log retrieval." - }, - { - "match": "or jobs history 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "file": "9b764201dd.json" - }, - { - "match": "or jobs traces 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "file": "d4abf1a83b.json", - "_doc": "Three recorded spans: RobotJob (full ConnectionHttpException stack in Attributes), the NewEmailReceived activity span (connector uipath-microsoft-outlook365, UseConnectionService=True), and the 'operation failed' error span." - }, - { - "match": "traces spans get --job-key 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "file": "d4abf1a83b.json", - "_doc": "Alternate trace-retrieval form (platform traces command). Same span payload as `or jobs traces` — without this rule the call falls through to the empty unmocked_default." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "437e071e60.json", - "_doc": "Faulted-only subset of the recorded folder job list (rows verbatim, filtered to State == Faulted)." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "16d8e9e6ea.json" - }, - { - "match": "or jobs list", - "file": "16d8e9e6ea.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "is connections list", - "file": "dabecc7dbd.json", - "exit_code": 1, - "_doc": "Live-probe evidence core to the diagnosis: the Integration Service connection layer was down when probed — HTTP 503 'no healthy upstream'. The original session probed three times (~35 min after the job) and received this byte-identical Failure payload on every attempt, so one representative fixture answers all probe retries. Matches any flag variation so repeated probes during the run keep seeing the outage. Failure response, so the CLI exits non-zero." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "min": 1, - "description": "Agent must fetch the failing job to read the ConnectionHttpException: Service Unavailable stack (HttpConnectionServiceClient.RequestWithRetryAsync exhausted, surfaced as Office365Exception) from the NewEmailReceived trigger activity." - }, - { - "pattern": "is connections list", - "min": 1, - "description": "Live probe of the Integration Service connection layer — the recorded HTTP 503 'no healthy upstream' response is the evidence that distinguishes a service-side 5xx outage from connection-deleted (404-class) or auth-revoked (401/403-class) causes." - }, - { - "pattern": "or jobs logs 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "min": 0, - "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "traces", - "min": 0, - "description": "OPTIONAL — spans confirm the NewEmailReceived activity (connector uipath-microsoft-outlook365, UseConnectionService=True) faulted with 'Service Unavailable'; the same stack is already fully visible in jobs get." - }, - { - "pattern": "or jobs history 1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/uip-fixture.json new file mode 100644 index 0000000000..b8598a4cec --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/data/uip-fixture.json @@ -0,0 +1,171 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 21,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1db8bac1-ecc2-4670-843e-ff5f9a26c532", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T19:32:28.579Z\",\r\n \"Message\": \"New Email Received: Service Unavailable\\r\\nin Main.xaml\\r\\n at NewEmailReceived \\\"New Email Received\\\"\\r\\n at Sequence \\\"O365 email trigger connection failure repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.MicrosoftOffice365.Office365Exception\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "1db8bac1-ecc2-4670-843e-ff5f9a26c532" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-629c-5e83d4c6a10f\",\r\n \"TraceId\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"StartTime\": \"2026-06-10T19:32:26.9708545Z\",\r\n \"EndTime\": \"2026-06-10T19:32:28.6336416Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.MicrosoftOffice365.Office365Exception: Service Unavailable\\\\r\\\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Service Unavailable\\\\r\\\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service Unavailable\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\\\r\\\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"a44a2ebf-2f3e-49f7-ad56-2601f66b3c65\",\r\n \"JobKey\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:32:28.9110662Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-c852-4be99f090254\",\r\n \"TraceId\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ParentId\": \"00000000-0000-0000-629c-5e83d4c6a10f\",\r\n \"Name\": \"New Email Received\",\r\n \"StartTime\": \"2026-06-10T19:32:27.0496037Z\",\r\n \"EndTime\": \"2026-06-10T19:32:27.0946303Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived\\\",\\\"Private.MailFolderArg\\\":{\\\"FolderName\\\":\\\"Inbox\\\",\\\"SelectionMode\\\":\\\"EnterPath\\\"},\\\"Private.BrowserFolderName\\\":\\\"Inbox\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Private.Result\\\":\\\"email\\\",\\\"Public.Connector\\\":\\\"uipath-microsoft-outlook365\\\",\\\"Public.WithAttachmentsOnly\\\":\\\"False\\\",\\\"Public.IncludeAttachments\\\":\\\"False\\\",\\\"Public.UseSharedMailbox\\\":\\\"False\\\",\\\"Private.MailboxArg\\\":{\\\"UseSharedMailbox\\\":\\\"False\\\"},\\\"Public.ConnectionId\\\":\\\"00000000-0000-0000-0000-000000000000\\\",\\\"Public.UseConnectionService\\\":\\\"True\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"New Email Received\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"a44a2ebf-2f3e-49f7-ad56-2601f66b3c65\",\r\n \"JobKey\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:32:28.0806145Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-e51f-b0044f677568\",\r\n \"TraceId\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ParentId\": \"00000000-0000-0000-c852-4be99f090254\",\r\n \"Name\": \"The operation failed with exception: 'Service Unavailable'\",\r\n \"StartTime\": \"2026-06-10T19:32:28.5632205Z\",\r\n \"EndTime\": \"2026-06-10T19:32:28.5637664Z\",\r\n \"Attributes\": \"{\\\"Private.Last requests\\\":\\\"\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"a44a2ebf-2f3e-49f7-ad56-2601f66b3c65\",\r\n \"JobKey\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:32:28.9110662Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 30,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "1db8bac1-ecc2-4670-843e-ff5f9a26c532" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"01a8cd00-0985-434a-8ffd-3a81a9fbad66\",\r\n \"Info\": \"Service Unavailable\\r\\n\\r\\nin Main.xaml\\r\\n at NewEmailReceived \\\"New Email Received\\\"\\r\\n at Sequence \\\"O365 email trigger connection failure repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.MicrosoftOffice365.Office365Exception: Service Unavailable\\r\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Service Unavailable\\r\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service Unavailable\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\r\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\r\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\",\r\n \"ReleaseName\": \"MailEventListener\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53473,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"1db8bac1ecc24670843eff5f9a26c532\",\r\n \"Id\": 4015114\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1db8bac1-ecc2-4670-843e-ff5f9a26c532" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T19:32:28.943Z\",\r\n \"Message\": \"MailEventListener execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T19:32:28.579Z\",\r\n \"Message\": \"New Email Received: Service Unavailable\\r\\nin Main.xaml\\r\\n at NewEmailReceived \\\"New Email Received\\\"\\r\\n at Sequence \\\"O365 email trigger connection failure repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.MicrosoftOffice365.Office365Exception\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T19:32:26.995Z\",\r\n \"Message\": \"MailEventListener execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "1db8bac1-ecc2-4670-843e-ff5f9a26c532" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-10T19:32:29.087Z\",\r\n \"JobId\": 4015114\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-10T19:32:26.620Z\",\r\n \"JobId\": 4015114\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-10T19:32:26.277Z\",\r\n \"JobId\": 4015114\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "1db8bac1-ecc2-4670-843e-ff5f9a26c532" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-629c-5e83d4c6a10f\",\r\n \"TraceId\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"StartTime\": \"2026-06-10T19:32:26.9708545Z\",\r\n \"EndTime\": \"2026-06-10T19:32:28.6336416Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.MicrosoftOffice365.Office365Exception: Service Unavailable\\\\r\\\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Service Unavailable\\\\r\\\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Service Unavailable\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\\\r\\\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetGraphConnectionAsync(CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.Models.BaseMailboxArgument.ResolveAsync(IMailboxArgValue value, IGraphServiceClientProxy service, CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived.ExecuteWithRetryAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Productivity.Retry.Retry.ExecuteWithTimeout[T](Func`1 action, Func`2 exceptionFilter, TimeSpan retryTimeout, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.BaseLeafConnectionServiceActivityWithoutBindings.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"a44a2ebf-2f3e-49f7-ad56-2601f66b3c65\",\r\n \"JobKey\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:32:28.9110662Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-c852-4be99f090254\",\r\n \"TraceId\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ParentId\": \"00000000-0000-0000-629c-5e83d4c6a10f\",\r\n \"Name\": \"New Email Received\",\r\n \"StartTime\": \"2026-06-10T19:32:27.0496037Z\",\r\n \"EndTime\": \"2026-06-10T19:32:27.0946303Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Mail.Triggers.NewEmailReceived\\\",\\\"Private.MailFolderArg\\\":{\\\"FolderName\\\":\\\"Inbox\\\",\\\"SelectionMode\\\":\\\"EnterPath\\\"},\\\"Private.BrowserFolderName\\\":\\\"Inbox\\\",\\\"Public.MarkAsRead\\\":\\\"False\\\",\\\"Private.Result\\\":\\\"email\\\",\\\"Public.Connector\\\":\\\"uipath-microsoft-outlook365\\\",\\\"Public.WithAttachmentsOnly\\\":\\\"False\\\",\\\"Public.IncludeAttachments\\\":\\\"False\\\",\\\"Public.UseSharedMailbox\\\":\\\"False\\\",\\\"Private.MailboxArg\\\":{\\\"UseSharedMailbox\\\":\\\"False\\\"},\\\"Public.ConnectionId\\\":\\\"00000000-0000-0000-0000-000000000000\\\",\\\"Public.UseConnectionService\\\":\\\"True\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"New Email Received\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"a44a2ebf-2f3e-49f7-ad56-2601f66b3c65\",\r\n \"JobKey\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:32:28.0806145Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n },\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-e51f-b0044f677568\",\r\n \"TraceId\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ParentId\": \"00000000-0000-0000-c852-4be99f090254\",\r\n \"Name\": \"The operation failed with exception: 'Service Unavailable'\",\r\n \"StartTime\": \"2026-06-10T19:32:28.5632205Z\",\r\n \"EndTime\": \"2026-06-10T19:32:28.5637664Z\",\r\n \"Attributes\": \"{\\\"Private.Last requests\\\":\\\"\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"a44a2ebf-2f3e-49f7-ad56-2601f66b3c65\",\r\n \"JobKey\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-10T19:32:28.9110662Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 30,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"HTTP 503: no healthy upstream\",\r\n \"Instructions\": \"Verify the connector key and folder key, then retry. Use 'uip is connections list --help' for accepted options.\\n\\nDetails: no healthy upstream\"\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 1db8bac1-ecc2-4670-843e-ff5f9a26c532", + "min": 1, + "description": "Agent must fetch the failing job to read the ConnectionHttpException: Service Unavailable stack (HttpConnectionServiceClient.RequestWithRetryAsync exhausted, surfaced as Office365Exception) from the NewEmailReceived trigger activity." + }, + { + "pattern": "is connections list", + "min": 1, + "description": "Live probe of the Integration Service connection layer — the recorded HTTP 503 'no healthy upstream' response is the evidence that distinguishes a service-side 5xx outage from connection-deleted (404-class) or auth-revoked (401/403-class) causes." + }, + { + "pattern": "or jobs logs 1db8bac1-ecc2-4670-843e-ff5f9a26c532", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "traces", + "min": 0, + "description": "OPTIONAL — spans confirm the NewEmailReceived activity (connector uipath-microsoft-outlook365, UseConnectionService=True) faulted with 'Service Unavailable'; the same stack is already fully visible in jobs get." + }, + { + "pattern": "or jobs history 1db8bac1-ecc2-4670-843e-ff5f9a26c532", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted) confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/task.yaml index baefc2680e..7513048c9e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-trigger-connection-503/task.yaml @@ -22,21 +22,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/README.md new file mode 100644 index 0000000000..4e30134013 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/README.md @@ -0,0 +1,95 @@ +# O365 Upload File — Per-File Size Limit Exceeded (maxFileSizeExceeded) — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session (scrubbed). + +## What the original session uncovered + +The user asked why their last job in folder `Shared` failed. The most recent +job, `DocumentUploader` (key `7fd5e890-f162-4591-a895-e649b09aea02`), +faulted at the legacy **Upload File** activity ("Upload File (point at +oversize file or near-full drive for quota/size errors)" in +`Main.xaml`) inside the Microsoft 365 Scope. Microsoft Graph +rejected the chunked upload at upload-session creation with a raw +`Microsoft.Graph.ServiceException`: `Code: invalidRequest`, `Message: The +payload of the request was too large`, `Inner error: Code: +maxFileSizeExceeded` — the documented over-size rejection: the file's +declared size exceeds the OneDrive/SharePoint per-file upload limit, +refused before any bytes transferred. Stack: +`LargeFileUploadTask.UploadAsync` / `UploadSliceRequest.PutAsync` / +`SimpleHttpProvider.SendAsync` into `UploadFile.ExecuteAsync`; the raw +`ServiceException` (no `Office365Exception` wrapper) marks the legacy code +path. Ruled out by message form: NOT a storage-quota problem +(`quotaLimitReached` absent), NOT a transient upload-session break, NOT +throttling. Deterministic on every run with the same file. Fix: compress or +split the file, or store it elsewhere and share a link; restart only after +the file is reduced. Playbook: +`references/activity-packages/o365-activities/playbooks/upload-file-quota-or-size.md`. + +### Evidence gap: traces unavailable + +In the real session job execution traces could NOT be retrieved — `uip +traces spans get --job-key 7fd5e890-...` returned `{"Result": "Failure", +"Message": "Error retrieving trace ID for job"}` with exit code 1. The +replay reproduces that failure faithfully: every trace-retrieval form +(`or jobs traces `, `traces spans get --job-key `, and the +TraceId-positional `traces spans get <32-hex>`) returns the recorded +failure payload with exit code 1. There is no span evidence for this job; +the diagnosis must come from `jobs get` + the Error-level job log alone — +both carry the full ServiceException with the cause-discriminating +`maxFileSizeExceeded` inner code. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the original investigation determined the root cause +from platform evidence alone (job Info and the Error-level log carry the full +Graph ServiceException and .NET stack). + +### Fixture derivations and scrubs + +- `or-jobs-logs-...-level-error.json` is the recorded Error-level log query + (the only logs raw captured); the generic logs rule maps to the same + fixture — the exception row is the only diagnostic content. +- `or-jobs-history-...-output-json.json` is the recorded verbatim history: + Pending → Running → Faulted in ~8 seconds, a clean single execution. +- No `--state Faulted` jobs-list raw was recorded, so there is no + Faulted-only fixture; the folder-key rule substring-matches + `--state Faulted` variants and returns the full recorded list. +- `traces-retrieval-error-...json` is the verbatim recorded trace-retrieval + failure (see Evidence gap above); all trace forms map to it with + exit code 1. The TraceId-positional form was not attempted in the real + session, but no span raw exists — serving the same recorded failure + preserves the evidence gap instead of fabricating an empty span set. +- The uploaded file's actual path and byte size were not in the platform + evidence (project source unavailable); Graph's rejection of the declared + size is the authoritative evidence. +- Standard scrubs applied: hostname → `MOCK-HOST`, workspace owner email → + `original_email@test.com`, Windows account → `UIPATH\REPLACEMENT_USER`. + Error text, stacks, keys, and timestamps are verbatim. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` + +## Re-running the extraction + +This scenario was hand-built from the investigation's `raw/` verbatim CLI +outputs. If the source investigation changes, update the fixture JSONs +directly and re-run the scrub checks, or regenerate with: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript --resolution \ + --scenario-name o365-upload-maxfilesize --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/1b284a08f9.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/1b284a08f9.json deleted file mode 100644 index ce082dc4fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/1b284a08f9.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/5c4780aa19.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/5c4780aa19.json deleted file mode 100644 index a192feed5a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/5c4780aa19.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7fd5e890-f162-4591-a895-e649b09aea02", - "StartTime": "2026-06-11T08:15:47.690Z", - "EndTime": "2026-06-11T08:15:54.983Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "258032d3-d0a5-489a-ada7-ca0858fb7927", - "Info": "Code: invalidRequest\r\nMessage: The payload of the request was too large\r\nInner error:\r\n\tCode: maxFileSizeExceeded\r\n\r\n\r\nin Main.xaml\r\n at UploadFile \"Upload File (point at oversize file or near-full drive for quota/size errors)\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Upload quota-or-size repro\"\r\n at Main \"Main\"\r\n\r\nMicrosoft.Graph.ServiceException: Code: invalidRequest\r\nMessage: The payload of the request was too large\r\nInner error:\r\n\tCode: maxFileSizeExceeded\r\n\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.UploadSliceRequest`1.SendRequestAsync(Stream stream, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.UploadSliceRequest`1.PutAsync(Stream stream, CancellationToken cancellationToken)\r\n at Microsoft.Graph.LargeFileUploadTask`1.UploadSliceAsync(UploadSliceRequest`1 uploadSliceRequest, ICollection`1 exceptionTrackingList)\r\n at Microsoft.Graph.LargeFileUploadTask`1.UploadAsync(IProgress`1 progress, Int32 maxTries)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceWithMetadataAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, Dictionary`2 driveItemMetadata, String account, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.UploadFileAsync(UploadRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Files.UploadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-11T08:15:46.967Z", - "ReleaseName": "DocumentUploader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53476, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T08:15:54.983Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7fd5e890f1624591a895e649b09aea02", - "Id": 4015118 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/86aed37609.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/86aed37609.json deleted file mode 100644 index c162d13077..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/86aed37609.json +++ /dev/null @@ -1,474 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7fd5e890-f162-4591-a895-e649b09aea02", - "State": "Faulted", - "ProcessName": "DocumentUploader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:15:47.690Z", - "EndTime": "2026-06-11T08:15:54.983Z", - "CreationTime": "2026-06-11T08:15:46.967Z" - }, - { - "Key": "e893a4d4-e061-4cba-a799-6ed4de6ee4e2", - "State": "Faulted", - "ProcessName": "TeamMailRetriever", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T08:01:51.017Z", - "EndTime": "2026-06-11T08:01:59.527Z", - "CreationTime": "2026-06-11T08:01:50.607Z" - }, - { - "Key": "cbbaaaf6-b416-4025-9a98-c8ee50990f76", - "State": "Faulted", - "ProcessName": "SupportMailLookup", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:45:13.687Z", - "EndTime": "2026-06-10T19:45:17.903Z", - "CreationTime": "2026-06-10T19:45:12.887Z" - }, - { - "Key": "1db8bac1-ecc2-4670-843e-ff5f9a26c532", - "State": "Faulted", - "ProcessName": "MailEventListener", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:32:26.610Z", - "EndTime": "2026-06-10T19:32:29.057Z", - "CreationTime": "2026-06-10T19:32:26.273Z" - }, - { - "Key": "64580780-4aa1-4ec2-b5f2-abd5439ea200", - "State": "Faulted", - "ProcessName": "ReportDownloader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T19:09:03.767Z", - "EndTime": "2026-06-10T19:09:09.557Z", - "CreationTime": "2026-06-10T19:09:03.457Z" - }, - { - "Key": "a04406a9-4c64-40d0-a7f3-042c3224c13b", - "State": "Faulted", - "ProcessName": "DriveFolderProvisioner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:59:36.937Z", - "EndTime": "2026-06-10T18:59:39.683Z", - "CreationTime": "2026-06-10T18:59:36.350Z" - }, - { - "Key": "4b4a58f8-6a7c-4af2-8700-1b6479b51028", - "State": "Faulted", - "ProcessName": "InboxQueryRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:37:23.003Z", - "EndTime": "2026-06-10T18:37:26.293Z", - "CreationTime": "2026-06-10T18:37:22.647Z" - }, - { - "Key": "74db8e32-8535-40b2-9b40-62540456adab", - "State": "Faulted", - "ProcessName": "NotificationMailer", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T18:15:34.930Z", - "EndTime": "2026-06-10T18:15:38.250Z", - "CreationTime": "2026-06-10T18:15:34.323Z" - }, - { - "Key": "50bcdb33-f7a8-42d4-a52d-ebbeeb135e99", - "State": "Faulted", - "ProcessName": "MailForwarder", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:58:41.673Z", - "EndTime": "2026-06-10T17:58:43.360Z", - "CreationTime": "2026-06-10T17:58:41.137Z" - }, - { - "Key": "909ed39c-08a1-48fb-921a-06d332f8f5fb", - "State": "Faulted", - "ProcessName": "DriveItemCopier", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T17:12:06.160Z", - "EndTime": "2026-06-10T17:12:14.247Z", - "CreationTime": "2026-06-10T16:53:17.843Z" - }, - { - "Key": "de11f5f7-a48a-4f76-bb30-ada33ea307fe", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:52:19.450Z", - "EndTime": "2026-06-09T12:52:22.867Z", - "CreationTime": "2026-06-09T12:52:19.450Z" - }, - { - "Key": "292c80e4-bc4f-41e3-9897-0fe73e0c64db", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T12:31:11.730Z", - "EndTime": "2026-06-09T12:31:13.000Z", - "CreationTime": "2026-06-09T12:31:11.730Z" - }, - { - "Key": "b7974725-4438-4315-a9cc-9ec41f8acd62", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-09T10:49:38.483Z", - "EndTime": "2026-06-09T10:50:41.847Z", - "CreationTime": "2026-06-09T10:49:38.483Z" - }, - { - "Key": "3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T15:01:56.683Z", - "EndTime": "2026-05-15T15:02:30.460Z", - "CreationTime": "2026-05-15T15:01:56.000Z" - }, - { - "Key": "701b6d1e-a23f-47ef-994a-c3235f1a60e9", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T14:17:18.843Z", - "EndTime": "2026-05-15T14:17:54.233Z", - "CreationTime": "2026-05-15T14:17:18.337Z" - }, - { - "Key": "764188b8-3192-4231-badb-e4469b54445a", - "State": "Stopped", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:12.317Z", - "EndTime": "2026-05-15T14:16:12.317Z", - "CreationTime": "2026-05-15T14:16:05.343Z" - }, - { - "Key": "b105e605-b832-47dc-8e49-afefeb09e529", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-15T14:16:17.043Z", - "EndTime": "2026-05-15T14:16:17.043Z", - "CreationTime": "2026-05-15T07:50:19.297Z" - }, - { - "Key": "6dceaaac-474f-462e-8170-3dddad4f1ba4", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:49:22.583Z", - "EndTime": "2026-05-15T07:49:25.873Z", - "CreationTime": "2026-05-15T07:49:22.127Z" - }, - { - "Key": "c322a61c-22cf-4184-9bb4-dce0c1634d51", - "State": "Faulted", - "ProcessName": "BlankProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:25.253Z", - "EndTime": "2026-05-15T07:44:08.090Z", - "CreationTime": "2026-05-15T07:42:24.730Z" - }, - { - "Key": "15edc166-3a30-4649-967f-cf03e4524783", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:42:00.257Z", - "EndTime": "2026-05-15T07:42:02.270Z", - "CreationTime": "2026-05-15T07:41:59.697Z" - }, - { - "Key": "6a85aefa-cebb-4658-a728-4c4f562097d9", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:40:01.807Z", - "EndTime": "2026-05-15T07:40:04.897Z", - "CreationTime": "2026-05-15T07:40:01.050Z" - }, - { - "Key": "3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:38:01.233Z", - "EndTime": "2026-05-15T07:38:01.233Z", - "CreationTime": "2026-05-15T07:37:48.550Z" - }, - { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "CreationTime": "2026-05-14T12:06:46.917Z" - }, - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - }, - { - "Key": "0f9d854e-221e-487d-bb2b-fcc76a7d7461", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T07:37:43.183Z", - "EndTime": "2026-05-15T07:38:32.633Z", - "CreationTime": "2026-05-14T11:07:13.593Z" - }, - { - "Key": "bad89b79-48c4-49ef-9de3-513df72ab69f", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T07:51:41.157Z", - "EndTime": "2026-05-14T07:52:15.300Z", - "CreationTime": "2026-05-14T07:51:41.157Z" - }, - { - "Key": "1c4fabc8-32cd-4761-ba14-8060e71cc7be", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-14T07:34:13.670Z", - "EndTime": "2026-05-14T07:34:13.670Z", - "CreationTime": "2026-05-14T07:28:38.110Z" - }, - { - "Key": "43e192cf-00d5-47f2-a6da-bac84beacfc8", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:52:13.243Z", - "EndTime": "2026-05-13T11:52:54.410Z", - "CreationTime": "2026-05-13T11:52:13.243Z" - }, - { - "Key": "27c4c11d-c04d-4705-8203-86e1a346dac3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:51:17.757Z", - "EndTime": "2026-05-13T11:51:18.447Z", - "CreationTime": "2026-05-13T11:51:17.757Z" - }, - { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "CreationTime": "2026-05-13T08:42:11.170Z" - }, - { - "Key": "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T07:32:22.603Z", - "EndTime": "2026-05-13T07:32:23.463Z", - "CreationTime": "2026-05-13T07:32:22.603Z" - }, - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 33, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/README.md deleted file mode 100644 index 273189b485..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/README.md +++ /dev/null @@ -1,94 +0,0 @@ -# O365 Upload File — Per-File Size Limit Exceeded (maxFileSizeExceeded) — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session (scrubbed). - -## What the original session uncovered - -The user asked why their last job in folder `Shared` failed. The most recent -job, `DocumentUploader` (key `7fd5e890-f162-4591-a895-e649b09aea02`), -faulted at the legacy **Upload File** activity ("Upload File (point at -oversize file or near-full drive for quota/size errors)" in -`Main.xaml`) inside the Microsoft 365 Scope. Microsoft Graph -rejected the chunked upload at upload-session creation with a raw -`Microsoft.Graph.ServiceException`: `Code: invalidRequest`, `Message: The -payload of the request was too large`, `Inner error: Code: -maxFileSizeExceeded` — the documented over-size rejection: the file's -declared size exceeds the OneDrive/SharePoint per-file upload limit, -refused before any bytes transferred. Stack: -`LargeFileUploadTask.UploadAsync` / `UploadSliceRequest.PutAsync` / -`SimpleHttpProvider.SendAsync` into `UploadFile.ExecuteAsync`; the raw -`ServiceException` (no `Office365Exception` wrapper) marks the legacy code -path. Ruled out by message form: NOT a storage-quota problem -(`quotaLimitReached` absent), NOT a transient upload-session break, NOT -throttling. Deterministic on every run with the same file. Fix: compress or -split the file, or store it elsewhere and share a link; restart only after -the file is reduced. Playbook: -`references/activity-packages/o365-activities/playbooks/upload-file-quota-or-size.md`. - -### Evidence gap: traces unavailable - -In the real session job execution traces could NOT be retrieved — `uip -traces spans get --job-key 7fd5e890-...` returned `{"Result": "Failure", -"Message": "Error retrieving trace ID for job"}` with exit code 1. The -replay reproduces that failure faithfully: every trace-retrieval form -(`or jobs traces `, `traces spans get --job-key `, and the -TraceId-positional `traces spans get <32-hex>`) returns the recorded -failure payload with exit code 1. There is no span evidence for this job; -the diagnosis must come from `jobs get` + the Error-level job log alone — -both carry the full ServiceException with the cause-discriminating -`maxFileSizeExceeded` inner code. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session's investigation `raw/` outputs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -No `process/` snapshot — the original investigation determined the root cause -from platform evidence alone (job Info and the Error-level log carry the full -Graph ServiceException and .NET stack). - -### Fixture derivations and scrubs - -- `or-jobs-logs-...-level-error.json` is the recorded Error-level log query - (the only logs raw captured); the generic logs rule maps to the same - fixture — the exception row is the only diagnostic content. -- `or-jobs-history-...-output-json.json` is the recorded verbatim history: - Pending → Running → Faulted in ~8 seconds, a clean single execution. -- No `--state Faulted` jobs-list raw was recorded, so there is no - Faulted-only fixture; the folder-key rule substring-matches - `--state Faulted` variants and returns the full recorded list. -- `traces-retrieval-error-...json` is the verbatim recorded trace-retrieval - failure (see Evidence gap above); all trace forms map to it with - exit code 1. The TraceId-positional form was not attempted in the real - session, but no span raw exists — serving the same recorded failure - preserves the evidence gap instead of fabricating an empty span set. -- The uploaded file's actual path and byte size were not in the platform - evidence (project source unavailable); Graph's rejection of the declared - size is the authoritative evidence. -- Standard scrubs applied: hostname → `MOCK-HOST`, workspace owner email → - `original_email@test.com`, Windows account → `UIPATH\REPLACEMENT_USER`. - Error text, stacks, keys, and timestamps are verbatim. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent's final response reaches the same root cause and fix as `RESOLUTION.md` - -## Re-running the extraction - -This scenario was hand-built from the investigation's `raw/` verbatim CLI -outputs. If the source investigation changes, update the fixture JSONs -directly and re-run the scrub checks, or regenerate with: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript --resolution \ - --scenario-name o365-upload-maxfilesize --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/d5f2a798de.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/d5f2a798de.json deleted file mode 100644 index e1a16e5173..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/d5f2a798de.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:15:54.013Z", - "Message": "Upload File (point at oversize file or near-full drive for quota/size errors): Code: invalidRequest\r\nMessage: The payload of the request was too large\r\nInner error:\r\n\tCode: maxFileSizeExceeded\r\n\r\nin Main.xaml\r\n at UploadFile \"Upload File (point at oversize file or near-full drive for quota/size errors)\"\r\n at Sequence \"Do\"\r\n at Office365ApplicationScope \"Microsoft 365 Scope\"\r\n at Sequence \"O365 Upload quota-or-size repro\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nMicrosoft.Graph.ServiceException\r\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\r\n at Microsoft.Graph.UploadSliceRequest`1.SendRequestAsync(Stream stream, CancellationToken cancellationToken, HttpCompletionOption completionOption)\r\n at Microsoft.Graph.UploadSliceRequest`1.PutAsync(Stream stream, CancellationToken cancellationToken)\r\n at Microsoft.Graph.LargeFileUploadTask`1.UploadSliceAsync(UploadSliceRequest`1 uploadSliceRequest, ICollection`1 exceptionTrackingList)\r\n at Microsoft.Graph.LargeFileUploadTask`1.UploadAsync(IProgress`1 progress, Int32 maxTries)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceWithMetadataAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, Dictionary`2 driveItemMetadata, String account, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.UploadFileAsync(UploadRequest request, CancellationToken cancellationToken)\r\n at UiPath.MicrosoftOffice365.Activities.Files.UploadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/e57a81dc57.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/e57a81dc57.json deleted file mode 100644 index 27a30a8a40..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/e57a81dc57.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-11T08:15:55.030Z", - "JobId": 4015118 - }, - { - "State": "Running", - "Timestamp": "2026-06-11T08:15:47.707Z", - "JobId": 4015118 - }, - { - "State": "Pending", - "Timestamp": "2026-06-11T08:15:46.970Z", - "JobId": 4015118 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/manifest.json deleted file mode 100644 index 2fc16fa168..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/m/r/manifest.json +++ /dev/null @@ -1,99 +0,0 @@ -{ - "version": 2, - "_doc": "Canned responses extracted from the original investigation's raw/ CLI outputs (scrubbed). Substring dispatch, first match wins — specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 7fd5e890-f162-4591-a895-e649b09aea02", - "file": "5c4780aa19.json" - }, - { - "match": "or jobs logs 7fd5e890-f162-4591-a895-e649b09aea02 --level Error", - "file": "d5f2a798de.json", - "_doc": "Recorded verbatim. The session captured only the Error-level log query; the single row is the job's one Error entry." - }, - { - "match": "or jobs logs 7fd5e890-f162-4591-a895-e649b09aea02", - "file": "d5f2a798de.json", - "_doc": "No unfiltered-logs raw was recorded; the Error-level set answers any logs variant — the exception row is the only diagnostic content." - }, - { - "match": "or jobs history 7fd5e890-f162-4591-a895-e649b09aea02", - "file": "e57a81dc57.json", - "_doc": "Recorded verbatim: Pending -> Running -> Faulted in ~8s, single clean execution, no retries." - }, - { - "match": "or jobs traces 7fd5e890-f162-4591-a895-e649b09aea02", - "file": "1b284a08f9.json", - "exit_code": 1, - "_doc": "Faithful replay of the recorded evidence gap: in the original session trace retrieval FAILED with 'Error retrieving trace ID for job' (exit 1). Every trace-retrieval form returns the same recorded failure so the eval agent faces the same gap — the diagnosis must come from jobs get + error logs." - }, - { - "match": "traces spans get --job-key 7fd5e890-f162-4591-a895-e649b09aea02", - "file": "1b284a08f9.json", - "exit_code": 1, - "_doc": "The form actually run in the original session; the recorded response is the failure payload above, exit code 1." - }, - { - "match": "traces spans get 7fd5e890f1624591a895e649b09aea02", - "file": "1b284a08f9.json", - "exit_code": 1, - "_doc": "TraceId-positional form (32-hex from the job record). Not attempted in the original session, but trace retrieval failed there and no span raw exists; serving the same recorded failure preserves the evidence gap instead of fabricating an empty span set via unmocked_default." - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "86aed37609.json", - "_doc": "No --state Faulted raw was recorded in this session, so there is no Faulted-only fixture; substring dispatch routes `--state Faulted` variants here too, returning the full recorded list — the failing job is the most recent row." - }, - { - "match": "or jobs list", - "file": "86aed37609.json", - "_doc": "Fallback for flag-order variations. The tenant has one folder of interest (Shared); the recorded folder-scoped job list is the correct answer for any jobs-list variant the agent tries." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get 7fd5e890-f162-4591-a895-e649b09aea02", - "min": 1, - "description": "Agent must fetch the failing job: the Info field carries the full Microsoft.Graph.ServiceException — Code: invalidRequest, 'The payload of the request was too large', Inner error Code: maxFileSizeExceeded — with the stack through LargeFileUploadTask.UploadAsync / UploadSliceRequest.PutAsync into legacy UploadFile.ExecuteAsync. This plus the error log is the entire evidence base (traces unavailable)." - }, - { - "pattern": "or jobs logs 7fd5e890-f162-4591-a895-e649b09aea02", - "min": 0, - "description": "OPTIONAL — Error-level log (1 entry) repeats the exception already visible in jobs get, anchored to activity 'Upload File' in Main.xaml." - }, - { - "pattern": "or jobs history 7fd5e890-f162-4591-a895-e649b09aea02", - "min": 0, - "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted, ~8s) confirm a clean single execution with no retries." - }, - { - "pattern": "traces", - "min": 0, - "description": "OPTIONAL — every trace-retrieval form replays the recorded failure 'Error retrieving trace ID for job' (exit 1). Traces were unavailable in the real session; no span evidence exists for this job." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/uip-fixture.json new file mode 100644 index 0000000000..6863cdc94e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "7fd5e890-f162-4591-a895-e649b09aea02", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:15:54.013Z\",\r\n \"Message\": \"Upload File (point at oversize file or near-full drive for quota/size errors): Code: invalidRequest\\r\\nMessage: The payload of the request was too large\\r\\nInner error:\\r\\n\\tCode: maxFileSizeExceeded\\r\\n\\r\\nin Main.xaml\\r\\n at UploadFile \\\"Upload File (point at oversize file or near-full drive for quota/size errors)\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Upload quota-or-size repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nMicrosoft.Graph.ServiceException\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.UploadSliceRequest`1.SendRequestAsync(Stream stream, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.UploadSliceRequest`1.PutAsync(Stream stream, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.LargeFileUploadTask`1.UploadSliceAsync(UploadSliceRequest`1 uploadSliceRequest, ICollection`1 exceptionTrackingList)\\r\\n at Microsoft.Graph.LargeFileUploadTask`1.UploadAsync(IProgress`1 progress, Int32 maxTries)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceWithMetadataAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, Dictionary`2 driveItemMetadata, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.UploadFileAsync(UploadRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.UploadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "7fd5e890-f162-4591-a895-e649b09aea02" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 33,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7fd5e890-f162-4591-a895-e649b09aea02" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"258032d3-d0a5-489a-ada7-ca0858fb7927\",\r\n \"Info\": \"Code: invalidRequest\\r\\nMessage: The payload of the request was too large\\r\\nInner error:\\r\\n\\tCode: maxFileSizeExceeded\\r\\n\\r\\n\\r\\nin Main.xaml\\r\\n at UploadFile \\\"Upload File (point at oversize file or near-full drive for quota/size errors)\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Upload quota-or-size repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nMicrosoft.Graph.ServiceException: Code: invalidRequest\\r\\nMessage: The payload of the request was too large\\r\\nInner error:\\r\\n\\tCode: maxFileSizeExceeded\\r\\n\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.UploadSliceRequest`1.SendRequestAsync(Stream stream, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.UploadSliceRequest`1.PutAsync(Stream stream, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.LargeFileUploadTask`1.UploadSliceAsync(UploadSliceRequest`1 uploadSliceRequest, ICollection`1 exceptionTrackingList)\\r\\n at Microsoft.Graph.LargeFileUploadTask`1.UploadAsync(IProgress`1 progress, Int32 maxTries)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceWithMetadataAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, Dictionary`2 driveItemMetadata, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.UploadFileAsync(UploadRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.UploadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\",\r\n \"ReleaseName\": \"DocumentUploader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53476,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7fd5e890f1624591a895e649b09aea02\",\r\n \"Id\": 4015118\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7fd5e890-f162-4591-a895-e649b09aea02" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:15:54.013Z\",\r\n \"Message\": \"Upload File (point at oversize file or near-full drive for quota/size errors): Code: invalidRequest\\r\\nMessage: The payload of the request was too large\\r\\nInner error:\\r\\n\\tCode: maxFileSizeExceeded\\r\\n\\r\\nin Main.xaml\\r\\n at UploadFile \\\"Upload File (point at oversize file or near-full drive for quota/size errors)\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at Office365ApplicationScope \\\"Microsoft 365 Scope\\\"\\r\\n at Sequence \\\"O365 Upload quota-or-size repro\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nMicrosoft.Graph.ServiceException\\r\\n at Microsoft.Graph.SimpleHttpProvider.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.UploadSliceRequest`1.SendRequestAsync(Stream stream, CancellationToken cancellationToken, HttpCompletionOption completionOption)\\r\\n at Microsoft.Graph.UploadSliceRequest`1.PutAsync(Stream stream, CancellationToken cancellationToken)\\r\\n at Microsoft.Graph.LargeFileUploadTask`1.UploadSliceAsync(UploadSliceRequest`1 uploadSliceRequest, ICollection`1 exceptionTrackingList)\\r\\n at Microsoft.Graph.LargeFileUploadTask`1.UploadAsync(IProgress`1 progress, Int32 maxTries)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadStreamAsync(GraphServiceClient graphClient, Stream stream, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, String account, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Files.Extensions.FilesExtensions.UploadResourceWithMetadataAsync(GraphServiceClient graphClient, IResource resource, DriveItem destination, String cloudFileName, ConflictBehavior conflictBehavior, Dictionary`2 driveItemMetadata, String account, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.UploadFileAsync(UploadRequest request, CancellationToken cancellationToken)\\r\\n at UiPath.MicrosoftOffice365.Activities.Files.UploadFile.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365BaseClientActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.MicrosoftOffice365.Activities.Office365ClientActivity`2.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "7fd5e890-f162-4591-a895-e649b09aea02" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-11T08:15:55.030Z\",\r\n \"JobId\": 4015118\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-11T08:15:47.707Z\",\r\n \"JobId\": 4015118\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-11T08:15:46.970Z\",\r\n \"JobId\": 4015118\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "7fd5e890-f162-4591-a895-e649b09aea02" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "7fd5e890f1624591a895e649b09aea02" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7fd5e890-f162-4591-a895-e649b09aea02\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocumentUploader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:15:47.690Z\",\r\n \"EndTime\": \"2026-06-11T08:15:54.983Z\",\r\n \"CreationTime\": \"2026-06-11T08:15:46.967Z\"\r\n },\r\n {\r\n \"Key\": \"e893a4d4-e061-4cba-a799-6ed4de6ee4e2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TeamMailRetriever\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:01:51.017Z\",\r\n \"EndTime\": \"2026-06-11T08:01:59.527Z\",\r\n \"CreationTime\": \"2026-06-11T08:01:50.607Z\"\r\n },\r\n {\r\n \"Key\": \"cbbaaaf6-b416-4025-9a98-c8ee50990f76\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SupportMailLookup\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:45:13.687Z\",\r\n \"EndTime\": \"2026-06-10T19:45:17.903Z\",\r\n \"CreationTime\": \"2026-06-10T19:45:12.887Z\"\r\n },\r\n {\r\n \"Key\": \"1db8bac1-ecc2-4670-843e-ff5f9a26c532\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailEventListener\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:32:26.610Z\",\r\n \"EndTime\": \"2026-06-10T19:32:29.057Z\",\r\n \"CreationTime\": \"2026-06-10T19:32:26.273Z\"\r\n },\r\n {\r\n \"Key\": \"64580780-4aa1-4ec2-b5f2-abd5439ea200\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportDownloader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T19:09:03.767Z\",\r\n \"EndTime\": \"2026-06-10T19:09:09.557Z\",\r\n \"CreationTime\": \"2026-06-10T19:09:03.457Z\"\r\n },\r\n {\r\n \"Key\": \"a04406a9-4c64-40d0-a7f3-042c3224c13b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveFolderProvisioner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:59:36.937Z\",\r\n \"EndTime\": \"2026-06-10T18:59:39.683Z\",\r\n \"CreationTime\": \"2026-06-10T18:59:36.350Z\"\r\n },\r\n {\r\n \"Key\": \"4b4a58f8-6a7c-4af2-8700-1b6479b51028\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboxQueryRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:37:23.003Z\",\r\n \"EndTime\": \"2026-06-10T18:37:26.293Z\",\r\n \"CreationTime\": \"2026-06-10T18:37:22.647Z\"\r\n },\r\n {\r\n \"Key\": \"74db8e32-8535-40b2-9b40-62540456adab\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NotificationMailer\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T18:15:34.930Z\",\r\n \"EndTime\": \"2026-06-10T18:15:38.250Z\",\r\n \"CreationTime\": \"2026-06-10T18:15:34.323Z\"\r\n },\r\n {\r\n \"Key\": \"50bcdb33-f7a8-42d4-a52d-ebbeeb135e99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MailForwarder\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:58:41.673Z\",\r\n \"EndTime\": \"2026-06-10T17:58:43.360Z\",\r\n \"CreationTime\": \"2026-06-10T17:58:41.137Z\"\r\n },\r\n {\r\n \"Key\": \"909ed39c-08a1-48fb-921a-06d332f8f5fb\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DriveItemCopier\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T17:12:06.160Z\",\r\n \"EndTime\": \"2026-06-10T17:12:14.247Z\",\r\n \"CreationTime\": \"2026-06-10T16:53:17.843Z\"\r\n },\r\n {\r\n \"Key\": \"de11f5f7-a48a-4f76-bb30-ada33ea307fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:52:19.450Z\",\r\n \"EndTime\": \"2026-06-09T12:52:22.867Z\",\r\n \"CreationTime\": \"2026-06-09T12:52:19.450Z\"\r\n },\r\n {\r\n \"Key\": \"292c80e4-bc4f-41e3-9897-0fe73e0c64db\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T12:31:11.730Z\",\r\n \"EndTime\": \"2026-06-09T12:31:13.000Z\",\r\n \"CreationTime\": \"2026-06-09T12:31:11.730Z\"\r\n },\r\n {\r\n \"Key\": \"b7974725-4438-4315-a9cc-9ec41f8acd62\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T10:49:38.483Z\",\r\n \"EndTime\": \"2026-06-09T10:50:41.847Z\",\r\n \"CreationTime\": \"2026-06-09T10:49:38.483Z\"\r\n },\r\n {\r\n \"Key\": \"3f6e6f3c-c1c9-4b1c-8ea9-0a0fc843581c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T15:01:56.683Z\",\r\n \"EndTime\": \"2026-05-15T15:02:30.460Z\",\r\n \"CreationTime\": \"2026-05-15T15:01:56.000Z\"\r\n },\r\n {\r\n \"Key\": \"701b6d1e-a23f-47ef-994a-c3235f1a60e9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:17:18.843Z\",\r\n \"EndTime\": \"2026-05-15T14:17:54.233Z\",\r\n \"CreationTime\": \"2026-05-15T14:17:18.337Z\"\r\n },\r\n {\r\n \"Key\": \"764188b8-3192-4231-badb-e4469b54445a\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"EndTime\": \"2026-05-15T14:16:12.317Z\",\r\n \"CreationTime\": \"2026-05-15T14:16:05.343Z\"\r\n },\r\n {\r\n \"Key\": \"b105e605-b832-47dc-8e49-afefeb09e529\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"EndTime\": \"2026-05-15T14:16:17.043Z\",\r\n \"CreationTime\": \"2026-05-15T07:50:19.297Z\"\r\n },\r\n {\r\n \"Key\": \"6dceaaac-474f-462e-8170-3dddad4f1ba4\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:49:22.583Z\",\r\n \"EndTime\": \"2026-05-15T07:49:25.873Z\",\r\n \"CreationTime\": \"2026-05-15T07:49:22.127Z\"\r\n },\r\n {\r\n \"Key\": \"c322a61c-22cf-4184-9bb4-dce0c1634d51\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:25.253Z\",\r\n \"EndTime\": \"2026-05-15T07:44:08.090Z\",\r\n \"CreationTime\": \"2026-05-15T07:42:24.730Z\"\r\n },\r\n {\r\n \"Key\": \"15edc166-3a30-4649-967f-cf03e4524783\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:42:00.257Z\",\r\n \"EndTime\": \"2026-05-15T07:42:02.270Z\",\r\n \"CreationTime\": \"2026-05-15T07:41:59.697Z\"\r\n },\r\n {\r\n \"Key\": \"6a85aefa-cebb-4658-a728-4c4f562097d9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:40:01.807Z\",\r\n \"EndTime\": \"2026-05-15T07:40:04.897Z\",\r\n \"CreationTime\": \"2026-05-15T07:40:01.050Z\"\r\n },\r\n {\r\n \"Key\": \"3af1f1ae-9a38-4abc-aac0-dfc28d9dc2a2\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"EndTime\": \"2026-05-15T07:38:01.233Z\",\r\n \"CreationTime\": \"2026-05-15T07:37:48.550Z\"\r\n },\r\n {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\"\r\n },\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n },\r\n {\r\n \"Key\": \"0f9d854e-221e-487d-bb2b-fcc76a7d7461\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T07:37:43.183Z\",\r\n \"EndTime\": \"2026-05-15T07:38:32.633Z\",\r\n \"CreationTime\": \"2026-05-14T11:07:13.593Z\"\r\n },\r\n {\r\n \"Key\": \"bad89b79-48c4-49ef-9de3-513df72ab69f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:51:41.157Z\",\r\n \"EndTime\": \"2026-05-14T07:52:15.300Z\",\r\n \"CreationTime\": \"2026-05-14T07:51:41.157Z\"\r\n },\r\n {\r\n \"Key\": \"1c4fabc8-32cd-4761-ba14-8060e71cc7be\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"EndTime\": \"2026-05-14T07:34:13.670Z\",\r\n \"CreationTime\": \"2026-05-14T07:28:38.110Z\"\r\n },\r\n {\r\n \"Key\": \"43e192cf-00d5-47f2-a6da-bac84beacfc8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:52:13.243Z\",\r\n \"EndTime\": \"2026-05-13T11:52:54.410Z\",\r\n \"CreationTime\": \"2026-05-13T11:52:13.243Z\"\r\n },\r\n {\r\n \"Key\": \"27c4c11d-c04d-4705-8203-86e1a346dac3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:51:17.757Z\",\r\n \"EndTime\": \"2026-05-13T11:51:18.447Z\",\r\n \"CreationTime\": \"2026-05-13T11:51:17.757Z\"\r\n },\r\n {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\"\r\n },\r\n {\r\n \"Key\": \"77dc53dd-5c45-4a77-b39f-a52e9e7ef163\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T07:32:22.603Z\",\r\n \"EndTime\": \"2026-05-13T07:32:23.463Z\",\r\n \"CreationTime\": \"2026-05-13T07:32:22.603Z\"\r\n },\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 33,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get 7fd5e890-f162-4591-a895-e649b09aea02", + "min": 1, + "description": "Agent must fetch the failing job: the Info field carries the full Microsoft.Graph.ServiceException — Code: invalidRequest, 'The payload of the request was too large', Inner error Code: maxFileSizeExceeded — with the stack through LargeFileUploadTask.UploadAsync / UploadSliceRequest.PutAsync into legacy UploadFile.ExecuteAsync. This plus the error log is the entire evidence base (traces unavailable)." + }, + { + "pattern": "or jobs logs 7fd5e890-f162-4591-a895-e649b09aea02", + "min": 0, + "description": "OPTIONAL — Error-level log (1 entry) repeats the exception already visible in jobs get, anchored to activity 'Upload File' in Main.xaml." + }, + { + "pattern": "or jobs history 7fd5e890-f162-4591-a895-e649b09aea02", + "min": 0, + "description": "OPTIONAL — state transitions (Pending -> Running -> Faulted, ~8s) confirm a clean single execution with no retries." + }, + { + "pattern": "traces", + "min": 0, + "description": "OPTIONAL — every trace-retrieval form replays the recorded failure 'Error retrieving trace ID for job' (exit 1). Traces were unavailable in the real session; no span evidence exists for this job." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/task.yaml index bfde269b6f..44199b1f00 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/o365-upload-maxfilesize/task.yaml @@ -28,21 +28,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/README.md new file mode 100644 index 0000000000..3ff5d40e80 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/README.md @@ -0,0 +1,30 @@ +# PDF Extract Page Range — invalid Range (read the page-range parse error, not the file/content) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the malformed-`Range` failure of `UiPath.PDF.Activities` Extract PDF Page Range. + +## What this exercises + +An `Extract PDF Page Range` ("Extract document pages") opens its input PDF successfully but faults parsing the `Range` argument. The job ends Faulted with `UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.`, thrown from `PageRange.ParsePageRangePart` via `ValidationHelper.ParseRange`. The agent must read the **invalid page-range signature** as the cause (the `Range` argument) — not blame a missing file (`FileNotFoundException`), the PDF content/encryption (`PdfException`), an out-of-bounds page (`Page range is incorrect.`), or a workflow-logic bug. The fix is to correct the `Range` to a valid expression. + +Signature captured verbatim from a local `UiPath.PDF.Activities` 4.3.0 run (`ExtractPDFPageRange` with `Range="abc"` against a valid PDF); the workflow name was neutralized so the agent must diagnose from the exception message. Maps to the [pdf-invalid-page-range](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-invalid-page-range.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, InvalidPageRangeException + ExtractPDFPageRange stack) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the page-range parse error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/515c254914.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/515c254914.json deleted file mode 100644 index fa4f592187..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/515c254914.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T10:02:14.560Z", - "Message": "Extract document pages: The input string 'abc' was not in a correct format.\r\nException Type: UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException\r\nin Main.xaml\r\n at ExtractPDFPageRange \"Extract document pages\"\r\n at Sequence \"Extract document pages\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.\r\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.ParsePageRangePart(String rangePart, Int32 pagesCount)\r\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.Parse(String pageRange, Int32 pagesCount)\r\n at UiPath.PDF.Activities.PDF.ValidationHelper.ParseRange(String range, Int32 pagesCount)\r\n at UiPath.PDF.Activities.PDF.ExtractPDFPageRange.Execute(CodeActivityContext context)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/60c2e8c8db.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/60c2e8c8db.json deleted file mode 100644 index 5e72733462..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/60c2e8c8db.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "StartTime": "2026-06-20T10:02:11.402Z", - "EndTime": "2026-06-20T10:02:14.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c2d3e4f5-a6b7-4c8d-9e0f-1a2b3c4d5e6f", - "Info": "Source: Extract document pages\r\nMessage: The input string 'abc' was not in a correct format.\r\nException Type: UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException\r\nin Main.xaml\r\n at ExtractPDFPageRange \"Extract document pages\"\r\n at Sequence \"Extract document pages\"\r\n at Main \"Main\"\r\n\r\nUiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.\r\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.ParsePageRangePart(String rangePart, Int32 pagesCount)\r\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.Parse(String pageRange, Int32 pagesCount)\r\n at UiPath.PDF.Activities.PDF.ValidationHelper.ParseRange(String range, Int32 pagesCount)\r\n at UiPath.PDF.Activities.PDF.ExtractPDFPageRange.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T10:02:10.220Z", - "ReleaseName": "MonthlyReporting", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108803, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T10:02:14.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3d1b8e609a2c4f5d8e7b1c2d3e4f5a6b", - "Id": 50614107 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/README.md deleted file mode 100644 index 9ca09e325f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# PDF Extract Page Range — invalid Range (read the page-range parse error, not the file/content) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the malformed-`Range` failure of `UiPath.PDF.Activities` Extract PDF Page Range. - -## What this exercises - -An `Extract PDF Page Range` ("Extract document pages") opens its input PDF successfully but faults parsing the `Range` argument. The job ends Faulted with `UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.`, thrown from `PageRange.ParsePageRangePart` via `ValidationHelper.ParseRange`. The agent must read the **invalid page-range signature** as the cause (the `Range` argument) — not blame a missing file (`FileNotFoundException`), the PDF content/encryption (`PdfException`), an out-of-bounds page (`Page range is incorrect.`), or a workflow-logic bug. The fix is to correct the `Range` to a valid expression. - -Signature captured verbatim from a local `UiPath.PDF.Activities` 4.3.0 run (`ExtractPDFPageRange` with `Range="abc"` against a valid PDF); the workflow name was neutralized so the agent must diagnose from the exception message. Maps to the [pdf-invalid-page-range](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-invalid-page-range.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, InvalidPageRangeException + ExtractPDFPageRange stack) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the page-range parse error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/e13b29f71c.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/e13b29f71c.json deleted file mode 100644 index 3b4b4505ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/e13b29f71c.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "State": "Faulted", - "StartTime": "2026-06-20T10:02:11.402Z", - "EndTime": "2026-06-20T10:02:14.880Z", - "ReleaseName": "MonthlyReporting", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50614107 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/manifest.json deleted file mode 100644 index de579f63f0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.PDF.Activities Extract PDF Page Range activity faults with UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException 'The input string '''abc''' was not in a correct format.' because the configured Range is malformed. The agent must read the invalid page-range signature as the cause (the Range argument), not blame the PDF content, a missing file, or a workflow-logic bug. Signature captured verbatim from a local UiPath.PDF.Activities 4.3.0 run; workflow name scrubbed. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "e13b29f71c.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "e13b29f71c.json" - }, - { - "match": "or jobs list", - "file": "e13b29f71c.json" - }, - { - "match": "or jobs get 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "file": "60c2e8c8db.json" - }, - { - "match": "or jobs logs 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b --level Error", - "file": "515c254914.json" - }, - { - "match": "or jobs logs 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "file": "515c254914.json" - }, - { - "match": "or jobs traces 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", - "min": 1, - "description": "Agent fetches the faulted job to read the InvalidPageRangeException signature from the Info / Error log." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/uip-fixture.json new file mode 100644 index 0000000000..a3d1a15c7b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T10:02:11.402Z\",\r\n \"EndTime\": \"2026-06-20T10:02:14.880Z\",\r\n \"ReleaseName\": \"MonthlyReporting\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50614107\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T10:02:14.560Z\",\r\n \"Message\": \"Extract document pages: The input string 'abc' was not in a correct format.\\r\\nException Type: UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException\\r\\nin Main.xaml\\r\\n at ExtractPDFPageRange \\\"Extract document pages\\\"\\r\\n at Sequence \\\"Extract document pages\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.\\r\\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.ParsePageRangePart(String rangePart, Int32 pagesCount)\\r\\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.Parse(String pageRange, Int32 pagesCount)\\r\\n at UiPath.PDF.Activities.PDF.ValidationHelper.ParseRange(String range, Int32 pagesCount)\\r\\n at UiPath.PDF.Activities.PDF.ExtractPDFPageRange.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T10:02:11.402Z\",\r\n \"EndTime\": \"2026-06-20T10:02:14.880Z\",\r\n \"ReleaseName\": \"MonthlyReporting\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50614107\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b\",\r\n \"StartTime\": \"2026-06-20T10:02:11.402Z\",\r\n \"EndTime\": \"2026-06-20T10:02:14.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c2d3e4f5-a6b7-4c8d-9e0f-1a2b3c4d5e6f\",\r\n \"Info\": \"Source: Extract document pages\\r\\nMessage: The input string 'abc' was not in a correct format.\\r\\nException Type: UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException\\r\\nin Main.xaml\\r\\n at ExtractPDFPageRange \\\"Extract document pages\\\"\\r\\n at Sequence \\\"Extract document pages\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.\\r\\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.ParsePageRangePart(String rangePart, Int32 pagesCount)\\r\\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.Parse(String pageRange, Int32 pagesCount)\\r\\n at UiPath.PDF.Activities.PDF.ValidationHelper.ParseRange(String range, Int32 pagesCount)\\r\\n at UiPath.PDF.Activities.PDF.ExtractPDFPageRange.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T10:02:10.220Z\",\r\n \"ReleaseName\": \"MonthlyReporting\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108803,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T10:02:14.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3d1b8e609a2c4f5d8e7b1c2d3e4f5a6b\",\r\n \"Id\": 50614107\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T10:02:14.560Z\",\r\n \"Message\": \"Extract document pages: The input string 'abc' was not in a correct format.\\r\\nException Type: UiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException\\r\\nin Main.xaml\\r\\n at ExtractPDFPageRange \\\"Extract document pages\\\"\\r\\n at Sequence \\\"Extract document pages\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.DocumentProcessing.Contracts.Extensions.InvalidPageRangeException: The input string 'abc' was not in a correct format.\\r\\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.ParsePageRangePart(String rangePart, Int32 pagesCount)\\r\\n at UiPath.DocumentProcessing.Contracts.Extensions.PageRange.Parse(String pageRange, Int32 pagesCount)\\r\\n at UiPath.PDF.Activities.PDF.ValidationHelper.ParseRange(String range, Int32 pagesCount)\\r\\n at UiPath.PDF.Activities.PDF.ExtractPDFPageRange.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T10:02:11.402Z\",\r\n \"EndTime\": \"2026-06-20T10:02:14.880Z\",\r\n \"ReleaseName\": \"MonthlyReporting\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50614107\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get 3d1b8e60-9a2c-4f5d-8e7b-1c2d3e4f5a6b", + "min": 1, + "description": "Agent fetches the faulted job to read the InvalidPageRangeException signature from the Info / Error log." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/task.yaml index 901b32d094..c5d1350dda 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-extract-page-range-invalid/task.yaml @@ -20,20 +20,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/README.md new file mode 100644 index 0000000000..48d3cb5abf --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/README.md @@ -0,0 +1,30 @@ +# PDF Read Text — corrupt / invalid PDF (read the parse error, not a missing file) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the corrupt-input-stream failure of `UiPath.PDF.Activities.ReadPDFText`. + +## What this exercises + +A `Read PDF Text` ("Read document text") finds and opens its input file but the Digitizer reader cannot parse the bytes. The job ends Faulted with `UiPath.PDF.PdfException: Invalid input stream`, raised from `UiPath.PDF.PdfReader..ctor`. The agent must read the **corrupt/unreadable-PDF signature** as the cause — not blame a missing file (`FileNotFoundException`), encryption (`PdfIncorrectPasswordException` / Manage-PDF-Password's `A password for the encrypted PDF file was not supplied`), OCR (`No OCR Engine assigned.`), an invalid page range (`InvalidPageRangeException`), or a workflow-logic bug. The fix is to replace the input with a valid PDF (and fix any upstream step that produced the bad file). + +Signature source: the `Invalid input stream` re-throw in `UiPath.PDF` `PDFReader` (verbatim string literal in the package source); workflow name neutralized. The companion file-not-found and invalid-range scenarios were live-confirmed via `uip rpa run`; this corrupt case uses the source-verbatim signature (a live confirmation run was blocked by a Studio host cache). Maps to the [pdf-corrupt-or-image-input](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-corrupt-or-image-input.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, PdfException: Invalid input stream) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the parse error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/03ae33dbb0.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/03ae33dbb0.json deleted file mode 100644 index d27ca85313..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/03ae33dbb0.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "StartTime": "2026-06-20T08:31:55.700Z", - "EndTime": "2026-06-20T08:31:58.214Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d4e5f6a7-b8c9-4d0e-1f2a-3b4c5d6e7f80", - "Info": "Source: Read document text\r\nMessage: Invalid input stream\r\nException Type: UiPath.PDF.PdfException\r\nin Main.xaml\r\n at ReadPDFText \"Read document text\"\r\n at Sequence \"Read document text\"\r\n at Main \"Main\"\r\n\r\nUiPath.PDF.PdfException: Invalid input stream\r\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\r\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-20T08:31:54.600Z", - "ReleaseName": "StatementsBatch", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108520, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T08:31:58.214Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "9e4c1a725b3d4e8fa1c23d4e5f6a7b80", - "Id": 50611540 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/3f9d8167db.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/3f9d8167db.json deleted file mode 100644 index f8f6a79f4d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/3f9d8167db.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T08:31:57.900Z", - "Message": "Read document text: Invalid input stream\r\nException Type: UiPath.PDF.PdfException\r\nin Main.xaml\r\n at ReadPDFText \"Read document text\"\r\n at Sequence \"Read document text\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.PDF.PdfException: Invalid input stream\r\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\r\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/873a94cecd.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/873a94cecd.json deleted file mode 100644 index d08b287161..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/873a94cecd.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "State": "Faulted", - "StartTime": "2026-06-20T08:31:55.700Z", - "EndTime": "2026-06-20T08:31:58.214Z", - "ReleaseName": "StatementsBatch", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50611540 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/README.md deleted file mode 100644 index 74196f907b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# PDF Read Text — corrupt / invalid PDF (read the parse error, not a missing file) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the corrupt-input-stream failure of `UiPath.PDF.Activities.ReadPDFText`. - -## What this exercises - -A `Read PDF Text` ("Read document text") finds and opens its input file but the Digitizer reader cannot parse the bytes. The job ends Faulted with `UiPath.PDF.PdfException: Invalid input stream`, raised from `UiPath.PDF.PdfReader..ctor`. The agent must read the **corrupt/unreadable-PDF signature** as the cause — not blame a missing file (`FileNotFoundException`), encryption (`PdfIncorrectPasswordException` / Manage-PDF-Password's `A password for the encrypted PDF file was not supplied`), OCR (`No OCR Engine assigned.`), an invalid page range (`InvalidPageRangeException`), or a workflow-logic bug. The fix is to replace the input with a valid PDF (and fix any upstream step that produced the bad file). - -Signature source: the `Invalid input stream` re-throw in `UiPath.PDF` `PDFReader` (verbatim string literal in the package source); workflow name neutralized. The companion file-not-found and invalid-range scenarios were live-confirmed via `uip rpa run`; this corrupt case uses the source-verbatim signature (a live confirmation run was blocked by a Studio host cache). Maps to the [pdf-corrupt-or-image-input](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-corrupt-or-image-input.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, PdfException: Invalid input stream) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the parse error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/manifest.json deleted file mode 100644 index 92b48a62a5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.PDF.Activities ReadPDFText activity faults at runtime with UiPath.PDF.PdfException 'Invalid input stream' because the input file is corrupt / not a valid PDF (the Digitizer reader rejects the byte stream). Signature taken verbatim from UiPath.PDF source (PDFReader re-throw of 'Invalid input stream'); workflow name neutralized. The agent must read the corrupt/unreadable-PDF signature, not blame a missing file, encryption, OCR, or workflow logic. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "873a94cecd.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "873a94cecd.json" - }, - { - "match": "or jobs list", - "file": "873a94cecd.json" - }, - { - "match": "or jobs get 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "file": "03ae33dbb0.json" - }, - { - "match": "or jobs logs 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80 --level Error", - "file": "3f9d8167db.json" - }, - { - "match": "or jobs logs 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "file": "3f9d8167db.json" - }, - { - "match": "or jobs traces 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", - "min": 1, - "description": "Agent fetches the faulted job to read the PdfException 'Invalid input stream' signature from the Info / Error log." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/uip-fixture.json new file mode 100644 index 0000000000..c63d03c169 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T08:31:55.700Z\",\r\n \"EndTime\": \"2026-06-20T08:31:58.214Z\",\r\n \"ReleaseName\": \"StatementsBatch\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50611540\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T08:31:57.900Z\",\r\n \"Message\": \"Read document text: Invalid input stream\\r\\nException Type: UiPath.PDF.PdfException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.PDF.PdfException: Invalid input stream\\r\\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T08:31:55.700Z\",\r\n \"EndTime\": \"2026-06-20T08:31:58.214Z\",\r\n \"ReleaseName\": \"StatementsBatch\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50611540\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80\",\r\n \"StartTime\": \"2026-06-20T08:31:55.700Z\",\r\n \"EndTime\": \"2026-06-20T08:31:58.214Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d4e5f6a7-b8c9-4d0e-1f2a-3b4c5d6e7f80\",\r\n \"Info\": \"Source: Read document text\\r\\nMessage: Invalid input stream\\r\\nException Type: UiPath.PDF.PdfException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.PDF.PdfException: Invalid input stream\\r\\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-20T08:31:54.600Z\",\r\n \"ReleaseName\": \"StatementsBatch\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108520,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T08:31:58.214Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"9e4c1a725b3d4e8fa1c23d4e5f6a7b80\",\r\n \"Id\": 50611540\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T08:31:57.900Z\",\r\n \"Message\": \"Read document text: Invalid input stream\\r\\nException Type: UiPath.PDF.PdfException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.PDF.PdfException: Invalid input stream\\r\\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T08:31:55.700Z\",\r\n \"EndTime\": \"2026-06-20T08:31:58.214Z\",\r\n \"ReleaseName\": \"StatementsBatch\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50611540\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get 9e4c1a72-5b3d-4e8f-a1c2-3d4e5f6a7b80", + "min": 1, + "description": "Agent fetches the faulted job to read the PdfException 'Invalid input stream' signature from the Info / Error log." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/task.yaml index f44d7f03e9..6954424822 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-corrupt/task.yaml @@ -19,20 +19,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/README.md new file mode 100644 index 0000000000..5cbda92c31 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/README.md @@ -0,0 +1,30 @@ +# PDF Read Text — encrypted PDF / wrong password (read the password signature, not a missing/corrupt file) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the encrypted-PDF failure of `UiPath.PDF.Activities.ReadPDFText`. + +## What this exercises + +A `Read PDF Text` ("Read document text") opens a password-protected PDF but no/wrong `Password` was supplied, so the reader faults. The job ends Faulted with `UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException: The password is incorrect.` (from `UiPath.PDF.PdfReader..ctor` via `ReadPDFText.GetFileReader`). The agent must read the **encrypted-PDF/password signature** as the cause — not blame a missing file (`FileNotFoundException`), a corrupt PDF (`Invalid input stream`), Manage-PDF-Password argument strings, or a workflow-logic bug. The fix is to set the correct `Password` on the activity. + +Signature captured **verbatim from a live `uip rpa` run**: a valid PDF was encrypted with `Manage PDF Password`, then `Read PDF Text` opened it with no password → `The password is incorrect.`. Workflow/process names neutralized; no real document. Maps to the [pdf-encrypted-or-wrong-password](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-encrypted-or-wrong-password.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, PdfException: The password is incorrect.) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the password error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/3427f5d6e4.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/3427f5d6e4.json deleted file mode 100644 index b3ae95fad5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/3427f5d6e4.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T07:55:14.400Z", - "Message": "Read document text: The password is incorrect.\r\nException Type: UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException\r\nin Main.xaml\r\n at ReadPDFText \"Read document text\"\r\n at Sequence \"Read document text\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException: The password is incorrect.\r\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\r\n at UiPath.PDF.Activities.ReadPDFText.GetFileReader(String fileName, String password)\r\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/68d4f00f88.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/68d4f00f88.json deleted file mode 100644 index 0d5810ad4f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/68d4f00f88.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "State": "Faulted", - "StartTime": "2026-06-20T07:55:12.240Z", - "EndTime": "2026-06-20T07:55:14.760Z", - "ReleaseName": "ContractsIntake", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50609921 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/README.md deleted file mode 100644 index 0978a20c98..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# PDF Read Text — encrypted PDF / wrong password (read the password signature, not a missing/corrupt file) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the encrypted-PDF failure of `UiPath.PDF.Activities.ReadPDFText`. - -## What this exercises - -A `Read PDF Text` ("Read document text") opens a password-protected PDF but no/wrong `Password` was supplied, so the reader faults. The job ends Faulted with `UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException: The password is incorrect.` (from `UiPath.PDF.PdfReader..ctor` via `ReadPDFText.GetFileReader`). The agent must read the **encrypted-PDF/password signature** as the cause — not blame a missing file (`FileNotFoundException`), a corrupt PDF (`Invalid input stream`), Manage-PDF-Password argument strings, or a workflow-logic bug. The fix is to set the correct `Password` on the activity. - -Signature captured **verbatim from a live `uip rpa` run**: a valid PDF was encrypted with `Manage PDF Password`, then `Read PDF Text` opened it with no password → `The password is incorrect.`. Workflow/process names neutralized; no real document. Maps to the [pdf-encrypted-or-wrong-password](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-encrypted-or-wrong-password.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, PdfException: The password is incorrect.) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the password error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/ad51ce00fc.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/ad51ce00fc.json deleted file mode 100644 index 32f4a8616a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/ad51ce00fc.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "StartTime": "2026-06-20T07:55:12.240Z", - "EndTime": "2026-06-20T07:55:14.760Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "a7b8c9d0-e1f2-4a3b-8c4d-5e6f7a8b9c0d", - "Info": "Source: Read document text\r\nMessage: The password is incorrect.\r\nException Type: UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException\r\nin Main.xaml\r\n at ReadPDFText \"Read document text\"\r\n at Sequence \"Read document text\"\r\n at Main \"Main\"\r\n\r\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\r\n at UiPath.PDF.Activities.ReadPDFText.GetFileReader(String fileName, String password)\r\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T07:55:11.100Z", - "ReleaseName": "ContractsIntake", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108340, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T07:55:14.760Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8b6d2f041a7c4e93b5d82c9f0a1b3e40", - "Id": 50609921 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/manifest.json deleted file mode 100644 index bc45b7fcd4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.PDF.Activities ReadPDFText on an encrypted PDF faults with UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException 'The password is incorrect.' (from PdfReader..ctor -> ReadPDFText.GetFileReader) because no/wrong Password was supplied for the password-protected PDF. Signature captured verbatim from a live uip rpa run; workflow/process names neutralized. The agent must read the encrypted-PDF/password signature as the cause, not blame a missing file (FileNotFoundException), corrupt content (Invalid input stream), or Manage-PDF-Password argument strings. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "68d4f00f88.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "68d4f00f88.json" - }, - { - "match": "or jobs list", - "file": "68d4f00f88.json" - }, - { - "match": "or jobs get 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "file": "ad51ce00fc.json" - }, - { - "match": "or jobs logs 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40 --level Error", - "file": "3427f5d6e4.json" - }, - { - "match": "or jobs logs 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "file": "3427f5d6e4.json" - }, - { - "match": "or jobs traces 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in Shared to find the failing job." - }, - { - "pattern": "or jobs get 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", - "min": 1, - "description": "Agent fetches the faulted job to read the PdfException 'The password is incorrect.' signature." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/uip-fixture.json new file mode 100644 index 0000000000..3d682614da --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T07:55:12.240Z\",\r\n \"EndTime\": \"2026-06-20T07:55:14.760Z\",\r\n \"ReleaseName\": \"ContractsIntake\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50609921\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T07:55:14.400Z\",\r\n \"Message\": \"Read document text: The password is incorrect.\\r\\nException Type: UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException: The password is incorrect.\\r\\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\\r\\n at UiPath.PDF.Activities.ReadPDFText.GetFileReader(String fileName, String password)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T07:55:12.240Z\",\r\n \"EndTime\": \"2026-06-20T07:55:14.760Z\",\r\n \"ReleaseName\": \"ContractsIntake\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50609921\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40\",\r\n \"StartTime\": \"2026-06-20T07:55:12.240Z\",\r\n \"EndTime\": \"2026-06-20T07:55:14.760Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"a7b8c9d0-e1f2-4a3b-8c4d-5e6f7a8b9c0d\",\r\n \"Info\": \"Source: Read document text\\r\\nMessage: The password is incorrect.\\r\\nException Type: UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\\r\\n at UiPath.PDF.Activities.ReadPDFText.GetFileReader(String fileName, String password)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T07:55:11.100Z\",\r\n \"ReleaseName\": \"ContractsIntake\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108340,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T07:55:14.760Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8b6d2f041a7c4e93b5d82c9f0a1b3e40\",\r\n \"Id\": 50609921\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T07:55:14.400Z\",\r\n \"Message\": \"Read document text: The password is incorrect.\\r\\nException Type: UiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.DocumentUnderstanding.Digitizer.Exceptions.PdfException: The password is incorrect.\\r\\n at UiPath.PDF.PdfReader..ctor(String filepath, String password)\\r\\n at UiPath.PDF.Activities.ReadPDFText.GetFileReader(String fileName, String password)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T07:55:12.240Z\",\r\n \"EndTime\": \"2026-06-20T07:55:14.760Z\",\r\n \"ReleaseName\": \"ContractsIntake\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50609921\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in Shared to find the failing job." + }, + { + "pattern": "or jobs get 8b6d2f04-1a7c-4e93-b5d8-2c9f0a1b3e40", + "min": 1, + "description": "Agent fetches the faulted job to read the PdfException 'The password is incorrect.' signature." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/task.yaml index 49e2eca9ca..b88e001fdf 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-encrypted/task.yaml @@ -21,20 +21,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/README.md new file mode 100644 index 0000000000..972567b864 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/README.md @@ -0,0 +1,30 @@ +# PDF Read Text — file not found (read the missing-file signature, not the PDF content) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the input-file-validation failure of `UiPath.PDF.Activities.ReadPDFText`. + +## What this exercises + +A `Read PDF Text` ("Read document text") faults at argument validation because the configured input PDF doesn't exist on the robot host. The job ends Faulted with `System.IO.FileNotFoundException: Could not find file "C:\RPA\Invoices\incoming\invoice_4471.pdf"`, thrown from `UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath` before the PDF is opened. The agent must read the **missing-file signature** as the cause (the input path) — not blame the PDF content/encryption (that would be a `UiPath.PDF.PdfException`), OCR (`Read PDF With OCR` only), a page range, or a workflow-logic bug. The fix is to make the file present at the configured path on the robot host (or correct the path). + +Signature captured verbatim from a local `UiPath.PDF.Activities` 4.3.0 run (`ReadPDFText` against a non-existent path); the workflow name and file path were neutralized/scrubbed so the agent must diagnose from the exception message, not the filename. Maps to the [pdf-file-not-found-or-not-pdf](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-file-not-found-or-not-pdf.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, FileNotFoundException + ReadPDFText stack) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the missing-file error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/25e85eab3d.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/25e85eab3d.json deleted file mode 100644 index c9a6673a2e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/25e85eab3d.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T09:14:05.310Z", - "Message": "Read document text: Could not find file \"C:\\RPA\\Invoices\\incoming\\invoice_4471.pdf\"\r\nException Type: System.IO.FileNotFoundException\r\nin Main.xaml\r\n at ReadPDFText \"Read document text\"\r\n at Sequence \"Read document text\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IO.FileNotFoundException: Could not find file \"C:\\RPA\\Invoices\\incoming\\invoice_4471.pdf\"\r\n at UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath(String filePath, String extension)\r\n at UiPath.PDF.Activities.ReadTextBase.<>c__DisplayClass33_0.b__2(String p, String _)\r\n at UiPath.PDF.Activities.PDF.ItemArgument`1.ResolveAsync(ArgValue value, Func`3 pathValidator, CancellationToken cancellationToken)\r\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/7288879bf0.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/7288879bf0.json deleted file mode 100644 index 2a425354dd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/7288879bf0.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "StartTime": "2026-06-20T09:14:02.118Z", - "EndTime": "2026-06-20T09:14:05.642Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b1c2d3e4-f5a6-4b7c-8d9e-0f1a2b3c4d5e", - "Info": "Source: Read document text\r\nMessage: Could not find file \"C:\\RPA\\Invoices\\incoming\\invoice_4471.pdf\"\r\nException Type: System.IO.FileNotFoundException\r\nin Main.xaml\r\n at ReadPDFText \"Read document text\"\r\n at Sequence \"Read document text\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileNotFoundException: Could not find file \"C:\\RPA\\Invoices\\incoming\\invoice_4471.pdf\"\r\n at UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath(String filePath, String extension)\r\n at UiPath.PDF.Activities.ReadTextBase.<>c__DisplayClass33_0.b__2(String p, String _)\r\n at UiPath.PDF.Activities.PDF.ItemArgument`1.ResolveAsync(ArgValue value, Func`3 pathValidator, CancellationToken cancellationToken)\r\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T09:14:01.004Z", - "ReleaseName": "InvoiceProcessing", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108742, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T09:14:05.642Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7a3e9f124c8b4d6e9a1f2b3c4d5e6f70", - "Id": 50613882 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/94f4dd3dba.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/94f4dd3dba.json deleted file mode 100644 index 65374cadcb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/94f4dd3dba.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "State": "Faulted", - "StartTime": "2026-06-20T09:14:02.118Z", - "EndTime": "2026-06-20T09:14:05.642Z", - "ReleaseName": "InvoiceProcessing", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50613882 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/README.md deleted file mode 100644 index 19298e5fad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# PDF Read Text — file not found (read the missing-file signature, not the PDF content) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the input-file-validation failure of `UiPath.PDF.Activities.ReadPDFText`. - -## What this exercises - -A `Read PDF Text` ("Read document text") faults at argument validation because the configured input PDF doesn't exist on the robot host. The job ends Faulted with `System.IO.FileNotFoundException: Could not find file "C:\RPA\Invoices\incoming\invoice_4471.pdf"`, thrown from `UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath` before the PDF is opened. The agent must read the **missing-file signature** as the cause (the input path) — not blame the PDF content/encryption (that would be a `UiPath.PDF.PdfException`), OCR (`Read PDF With OCR` only), a page range, or a workflow-logic bug. The fix is to make the file present at the configured path on the robot host (or correct the path). - -Signature captured verbatim from a local `UiPath.PDF.Activities` 4.3.0 run (`ReadPDFText` against a non-existent path); the workflow name and file path were neutralized/scrubbed so the agent must diagnose from the exception message, not the filename. Maps to the [pdf-file-not-found-or-not-pdf](../../../../../skills/uipath-troubleshoot/references/activity-packages/pdf-activities/playbooks/pdf-file-not-found-or-not-pdf.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, FileNotFoundException + ReadPDFText stack) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the missing-file error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/manifest.json deleted file mode 100644 index 725e5c62fe..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.PDF.Activities ReadPDFText activity faults at argument validation with System.IO.FileNotFoundException 'Could not find file \"\"' because the input PDF path does not exist on the robot host. The agent must read the missing-file signature as the cause (the input path), not blame the PDF content, OCR, or a workflow-logic bug. Signature captured verbatim from a local UiPath.PDF.Activities 4.3.0 run; workflow name and path scrubbed/neutralized. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "94f4dd3dba.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "94f4dd3dba.json" - }, - { - "match": "or jobs list", - "file": "94f4dd3dba.json" - }, - { - "match": "or jobs get 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "file": "7288879bf0.json" - }, - { - "match": "or jobs logs 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70 --level Error", - "file": "25e85eab3d.json" - }, - { - "match": "or jobs logs 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "file": "25e85eab3d.json" - }, - { - "match": "or jobs traces 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", - "min": 1, - "description": "Agent fetches the faulted job to read the FileNotFoundException 'Could not find file' signature from the Info / Error log." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..1373da2ae6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T09:14:02.118Z\",\r\n \"EndTime\": \"2026-06-20T09:14:05.642Z\",\r\n \"ReleaseName\": \"InvoiceProcessing\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50613882\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T09:14:05.310Z\",\r\n \"Message\": \"Read document text: Could not find file \\\"C:\\\\RPA\\\\Invoices\\\\incoming\\\\invoice_4471.pdf\\\"\\r\\nException Type: System.IO.FileNotFoundException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException: Could not find file \\\"C:\\\\RPA\\\\Invoices\\\\incoming\\\\invoice_4471.pdf\\\"\\r\\n at UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath(String filePath, String extension)\\r\\n at UiPath.PDF.Activities.ReadTextBase.<>c__DisplayClass33_0.b__2(String p, String _)\\r\\n at UiPath.PDF.Activities.PDF.ItemArgument`1.ResolveAsync(ArgValue value, Func`3 pathValidator, CancellationToken cancellationToken)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T09:14:02.118Z\",\r\n \"EndTime\": \"2026-06-20T09:14:05.642Z\",\r\n \"ReleaseName\": \"InvoiceProcessing\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50613882\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70\",\r\n \"StartTime\": \"2026-06-20T09:14:02.118Z\",\r\n \"EndTime\": \"2026-06-20T09:14:05.642Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b1c2d3e4-f5a6-4b7c-8d9e-0f1a2b3c4d5e\",\r\n \"Info\": \"Source: Read document text\\r\\nMessage: Could not find file \\\"C:\\\\RPA\\\\Invoices\\\\incoming\\\\invoice_4471.pdf\\\"\\r\\nException Type: System.IO.FileNotFoundException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Could not find file \\\"C:\\\\RPA\\\\Invoices\\\\incoming\\\\invoice_4471.pdf\\\"\\r\\n at UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath(String filePath, String extension)\\r\\n at UiPath.PDF.Activities.ReadTextBase.<>c__DisplayClass33_0.b__2(String p, String _)\\r\\n at UiPath.PDF.Activities.PDF.ItemArgument`1.ResolveAsync(ArgValue value, Func`3 pathValidator, CancellationToken cancellationToken)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T09:14:01.004Z\",\r\n \"ReleaseName\": \"InvoiceProcessing\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108742,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T09:14:05.642Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7a3e9f124c8b4d6e9a1f2b3c4d5e6f70\",\r\n \"Id\": 50613882\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T09:14:05.310Z\",\r\n \"Message\": \"Read document text: Could not find file \\\"C:\\\\RPA\\\\Invoices\\\\incoming\\\\invoice_4471.pdf\\\"\\r\\nException Type: System.IO.FileNotFoundException\\r\\nin Main.xaml\\r\\n at ReadPDFText \\\"Read document text\\\"\\r\\n at Sequence \\\"Read document text\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.FileNotFoundException: Could not find file \\\"C:\\\\RPA\\\\Invoices\\\\incoming\\\\invoice_4471.pdf\\\"\\r\\n at UiPath.PDF.Activities.PDF.ValidationHelper.ValidateInputFilePath(String filePath, String extension)\\r\\n at UiPath.PDF.Activities.ReadTextBase.<>c__DisplayClass33_0.b__2(String p, String _)\\r\\n at UiPath.PDF.Activities.PDF.ItemArgument`1.ResolveAsync(ArgValue value, Func`3 pathValidator, CancellationToken cancellationToken)\\r\\n at UiPath.PDF.Activities.ReadTextBase.Execute(CodeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T09:14:02.118Z\",\r\n \"EndTime\": \"2026-06-20T09:14:05.642Z\",\r\n \"ReleaseName\": \"InvoiceProcessing\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50613882\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get 7a3e9f12-4c8b-4d6e-9a1f-2b3c4d5e6f70", + "min": 1, + "description": "Agent fetches the faulted job to read the FileNotFoundException 'Could not find file' signature from the Info / Error log." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/task.yaml index b397af75f4..954cadf7d5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/pdf-read-file-not-found/task.yaml @@ -19,20 +19,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/README.md new file mode 100644 index 0000000000..4a3fa18ea1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/README.md @@ -0,0 +1,54 @@ +# Invoke Python Method Failure - Function Not Bound at Module Level + +This scenario reproduces an `Invoke Python Method` +(`UiPath.Python.Activities.InvokeMethod`) failure. The Python engine +initializes and `Load Python Script` succeeds, then the invoke step faults +with `AttributeError: module 'transform' has no attribute 'clean_records'` +because the target function is defined only inside the script's +`if __name__ == "__main__":` block. + +## What this scenario uncovers + +**Root Cause:** `Load Python Script` binds functions defined at **module +level** and skips the `if __name__ == "__main__":` block on import. In +`scripts/transform.py`, `clean_records` is defined inside that guard, so it +is never bound and `Invoke Python Method` cannot resolve it. The script +"works standalone" (`python transform.py` runs the `__main__` block) but +fails under UiPath (which imports the module instead of running the file). + +This maps to: +`references/activity-packages/python-activities/playbooks/invoke-method-failures.md` +(sub-cause M1 - function name does not resolve at module level). + +The user is framed as **off-host** (Orchestrator only). The fix is a +script-structure change in the project source the agent can recommend (and +optionally apply) without host access; the correct behavior is to identify +the invoke-resolution fault and recommend moving the function to module +level - not to attempt host commands or misdiagnose it as an engine-init, +load, or argument failure. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project: `Python Scope` -> `Load Python Script` -> `Invoke Python Method`, with `clean_records` defined only under `__main__` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. The job logs deliberately show the +> engine initializing and the script loading BEFORE the fault, to +> distinguish this from a load-script-failures issue. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `invoke-method-failures.md` and identified that the function + is not bound at module level (defined under `__main__`), NOT an engine-init, + script-load, or argument-mismatch failure +- Agent recommended moving `clean_records` to module level (and/or validating + by `python -c "import transform; print(transform.clean_records)"`), without + fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/272c82aba7.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/272c82aba7.json deleted file mode 100644 index 8de2bb3153..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/272c82aba7.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:21:42.780Z", - "Message": "[Invoke Python Method] One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:21:42.820Z", - "Message": "One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at InvokeMethod \"Invoke Python Method\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/README.md deleted file mode 100644 index 6b5f42b434..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Invoke Python Method Failure - Function Not Bound at Module Level - -This scenario reproduces an `Invoke Python Method` -(`UiPath.Python.Activities.InvokeMethod`) failure. The Python engine -initializes and `Load Python Script` succeeds, then the invoke step faults -with `AttributeError: module 'transform' has no attribute 'clean_records'` -because the target function is defined only inside the script's -`if __name__ == "__main__":` block. - -## What this scenario uncovers - -**Root Cause:** `Load Python Script` binds functions defined at **module -level** and skips the `if __name__ == "__main__":` block on import. In -`scripts/transform.py`, `clean_records` is defined inside that guard, so it -is never bound and `Invoke Python Method` cannot resolve it. The script -"works standalone" (`python transform.py` runs the `__main__` block) but -fails under UiPath (which imports the module instead of running the file). - -This maps to: -`references/activity-packages/python-activities/playbooks/invoke-method-failures.md` -(sub-cause M1 - function name does not resolve at module level). - -The user is framed as **off-host** (Orchestrator only). The fix is a -script-structure change in the project source the agent can recommend (and -optionally apply) without host access; the correct behavior is to identify -the invoke-resolution fault and recommend moving the function to module -level - not to attempt host commands or misdiagnose it as an engine-init, -load, or argument failure. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project: `Python Scope` -> `Load Python Script` -> `Invoke Python Method`, with `clean_records` defined only under `__main__` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. The job logs deliberately show the -> engine initializing and the script loading BEFORE the fault, to -> distinguish this from a load-script-failures issue. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `invoke-method-failures.md` and identified that the function - is not bound at module level (defined under `__main__`), NOT an engine-init, - script-load, or argument-mismatch failure -- Agent recommended moving `clean_records` to module level (and/or validating - by `python -c "import transform; print(transform.clean_records)"`), without - fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/b7cca32729.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/b7cca32729.json deleted file mode 100644 index f628cb105a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/b7cca32729.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", - "StartTime": "2026-06-11T08:21:40.110Z", - "EndTime": "2026-06-11T08:21:42.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "e5f6a7b8-c9d0-4415-b6c7-d8e9f0123456", - "Info": "Invoke Python Method: One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at InvokeMethod \"Invoke Python Method\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'", - "CreationTime": "2026-06-11T08:21:39.700Z", - "ReleaseName": "PyRecordCleaner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7113441, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T08:21:42.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", - "Id": 713551298 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/d03abaad81.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/d03abaad81.json deleted file mode 100644 index 6e225db653..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/d03abaad81.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", - "Name": "Data Pipelines", - "Path": "Data Pipelines", - "Description": "Production folder for data-transformation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/eb211fb7f0.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/eb211fb7f0.json deleted file mode 100644 index 63496391a8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/eb211fb7f0.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-11T08:21:40.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T08:21:41.520Z", - "Message": "[Python Scope] Python engine initialized (Target: x64, Path: C:\\Python311)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T08:21:41.890Z", - "Message": "[Load Python Script] Loaded script: scripts\\transform.py" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:21:42.780Z", - "Message": "[Invoke Python Method] One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:21:42.820Z", - "Message": "One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at InvokeMethod \"Invoke Python Method\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T08:21:42.860Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/efcc674635.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/efcc674635.json deleted file mode 100644 index 8b6a5cca30..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/efcc674635.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", - "State": "Faulted", - "ProcessName": "PyRecordCleaner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-11T08:21:40.110Z", - "EndTime": "2026-06-11T08:21:42.880Z", - "CreationTime": "2026-06-11T08:21:39.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/manifest.json deleted file mode 100644 index db608ef496..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the python-invoke-method-name-unresolved scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "d03abaad81.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\" --state Faulted --output json", - "file": "efcc674635.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902 --state Faulted --output json", - "file": "efcc674635.json" - }, - { - "match": "or jobs get \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\" --output json", - "file": "b7cca32729.json" - }, - { - "match": "or jobs get d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345 --output json", - "file": "b7cca32729.json" - }, - { - "match": "or jobs logs \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\" --level Error --output json", - "file": "272c82aba7.json" - }, - { - "match": "or jobs logs d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345 --level Error --output json", - "file": "272c82aba7.json" - }, - { - "match": "or jobs logs \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\" --output json", - "file": "eb211fb7f0.json" - }, - { - "match": "or jobs logs d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345 --output json", - "file": "eb211fb7f0.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/uip-fixture.json new file mode 100644 index 0000000000..4999de4bb1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyRecordCleaner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:21:40.110Z\",\r\n \"EndTime\": \"2026-06-11T08:21:42.880Z\",\r\n \"CreationTime\": \"2026-06-11T08:21:39.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:21:42.780Z\",\r\n \"Message\": \"[Invoke Python Method] One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:21:42.820Z\",\r\n \"Message\": \"One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at InvokeMethod \\\"Invoke Python Method\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\",\r\n \"StartTime\": \"2026-06-11T08:21:40.110Z\",\r\n \"EndTime\": \"2026-06-11T08:21:42.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"e5f6a7b8-c9d0-4415-b6c7-d8e9f0123456\",\r\n \"Info\": \"Invoke Python Method: One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at InvokeMethod \\\"Invoke Python Method\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\",\r\n \"CreationTime\": \"2026-06-11T08:21:39.700Z\",\r\n \"ReleaseName\": \"PyRecordCleaner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7113441,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T08:21:42.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\",\r\n \"Id\": 713551298\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T08:21:40.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T08:21:41.520Z\",\r\n \"Message\": \"[Python Scope] Python engine initialized (Target: x64, Path: C:\\\\Python311)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T08:21:41.890Z\",\r\n \"Message\": \"[Load Python Script] Loaded script: scripts\\\\transform.py\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:21:42.780Z\",\r\n \"Message\": \"[Invoke Python Method] One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:21:42.820Z\",\r\n \"Message\": \"One or more errors occurred. ---> Python.Runtime.PythonException: AttributeError : module 'transform' has no attribute 'clean_records'\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at InvokeMethod \\\"Invoke Python Method\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T08:21:42.860Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\",\r\n \"Name\": \"Data Pipelines\",\r\n \"Path\": \"Data Pipelines\",\r\n \"Description\": \"Production folder for data-transformation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/task.yaml index 1d6176cd5f..cd8c0a54f6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-invoke-method-name-unresolved/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/README.md new file mode 100644 index 0000000000..33f1704133 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/README.md @@ -0,0 +1,49 @@ +# Load Python Script Failure - Python Engine Fails to Initialize + +This scenario reproduces a `Load Python Script` (`UiPath.Python.Activities.LoadScript`) +failure caused by the parent `Python Scope` being unable to **initialize +the Python engine** on the robot host. The job faults at startup with +`Error initializing Python engine 64 bit` before the script body loads. + +## What this scenario uncovers + +**Root Cause:** The `Python Scope` targets Python 3.11 (`Path=C:\Python311`, +`Version=Auto`, `Target=x64`) but its **`Library path` is empty**. On +Windows, for Python **> 3.9** the `Library path` must point at the matching +`pythonXX.dll` (`python311.dll`); left empty, the engine cannot initialize, +so `Load Python Script` faults before loading the script. A bitness +mismatch between the installed interpreter and `Target = x64` produces the +same error and must be ruled out on the host. + +This maps to: +`references/activity-packages/python-activities/playbooks/load-script-failures.md` +(engine-initialization family, sub-causes L1c `Library path` and L1b bitness). + +The user is framed as **off-host** (Orchestrator only), so the correct +agent behavior is to identify the engine-init failure, recommend setting +the `Library path` for Python > 3.9 (and/or verifying interpreter bitness), +and hand over the host checks - not to attempt host commands itself. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Python Scope` + `Load Python Script`, `Library path` left empty | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `load-script-failures.md` and identified an engine-initialization + failure (not a `ModuleNotFoundError` or syntax error) +- Agent recommended setting the `Library path` to the matching `pythonXX.dll` + for Python > 3.9, and/or verifying the interpreter bitness against + `Target = x64` (any valid engine-init fix path scores full marks), + without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/5e4fb795b8.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/5e4fb795b8.json deleted file mode 100644 index 7f3a0d8cae..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/5e4fb795b8.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T14:03:13.560Z", - "Message": "[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T14:03:13.600Z", - "Message": "One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/67696b56ff.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/67696b56ff.json deleted file mode 100644 index 89b43df126..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/67696b56ff.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123", - "StartTime": "2026-06-10T14:03:11.210Z", - "EndTime": "2026-06-10T14:03:13.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345", - "Info": "Load Python Script: One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit", - "CreationTime": "2026-06-10T14:03:10.800Z", - "ReleaseName": "PyDataTransform", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012332, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T14:03:13.640Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123", - "Id": 712440187 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/7d574be68e.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/7d574be68e.json deleted file mode 100644 index 19386b7682..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/7d574be68e.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-10T14:03:11.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-10T14:03:12.510Z", - "Message": "[Python Scope] Initializing Python engine (Target: x64, Path: C:\\Python311, Version: Auto)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T14:03:13.560Z", - "Message": "[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T14:03:13.600Z", - "Message": "One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-10T14:03:13.620Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/README.md deleted file mode 100644 index ad828d4ea2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Load Python Script Failure - Python Engine Fails to Initialize - -This scenario reproduces a `Load Python Script` (`UiPath.Python.Activities.LoadScript`) -failure caused by the parent `Python Scope` being unable to **initialize -the Python engine** on the robot host. The job faults at startup with -`Error initializing Python engine 64 bit` before the script body loads. - -## What this scenario uncovers - -**Root Cause:** The `Python Scope` targets Python 3.11 (`Path=C:\Python311`, -`Version=Auto`, `Target=x64`) but its **`Library path` is empty**. On -Windows, for Python **> 3.9** the `Library path` must point at the matching -`pythonXX.dll` (`python311.dll`); left empty, the engine cannot initialize, -so `Load Python Script` faults before loading the script. A bitness -mismatch between the installed interpreter and `Target = x64` produces the -same error and must be ruled out on the host. - -This maps to: -`references/activity-packages/python-activities/playbooks/load-script-failures.md` -(engine-initialization family, sub-causes L1c `Library path` and L1b bitness). - -The user is framed as **off-host** (Orchestrator only), so the correct -agent behavior is to identify the engine-init failure, recommend setting -the `Library path` for Python > 3.9 (and/or verifying interpreter bitness), -and hand over the host checks - not to attempt host commands itself. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Python Scope` + `Load Python Script`, `Library path` left empty | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `load-script-failures.md` and identified an engine-initialization - failure (not a `ModuleNotFoundError` or syntax error) -- Agent recommended setting the `Library path` to the matching `pythonXX.dll` - for Python > 3.9, and/or verifying the interpreter bitness against - `Target = x64` (any valid engine-init fix path scores full marks), - without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/d03abaad81.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/d03abaad81.json deleted file mode 100644 index 6e225db653..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/d03abaad81.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", - "Name": "Data Pipelines", - "Path": "Data Pipelines", - "Description": "Production folder for data-transformation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/d7c7a0328a.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/d7c7a0328a.json deleted file mode 100644 index f0d30799d0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/d7c7a0328a.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123", - "State": "Faulted", - "ProcessName": "PyDataTransform", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-10T14:03:11.210Z", - "EndTime": "2026-06-10T14:03:13.640Z", - "CreationTime": "2026-06-10T14:03:10.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/manifest.json deleted file mode 100644 index cd092fe0a4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the python-loadscript-engine-init scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "d03abaad81.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\" --state Faulted --output json", - "file": "d7c7a0328a.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902 --state Faulted --output json", - "file": "d7c7a0328a.json" - }, - { - "match": "or jobs get \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123\" --output json", - "file": "67696b56ff.json" - }, - { - "match": "or jobs get c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123 --output json", - "file": "67696b56ff.json" - }, - { - "match": "or jobs logs \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123\" --level Error --output json", - "file": "5e4fb795b8.json" - }, - { - "match": "or jobs logs c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123 --level Error --output json", - "file": "5e4fb795b8.json" - }, - { - "match": "or jobs logs \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123\" --output json", - "file": "7d574be68e.json" - }, - { - "match": "or jobs logs c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123 --output json", - "file": "7d574be68e.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/uip-fixture.json new file mode 100644 index 0000000000..b66873722d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyDataTransform\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T14:03:11.210Z\",\r\n \"EndTime\": \"2026-06-10T14:03:13.640Z\",\r\n \"CreationTime\": \"2026-06-10T14:03:10.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T14:03:13.560Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T14:03:13.600Z\",\r\n \"Message\": \"One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123\",\r\n \"StartTime\": \"2026-06-10T14:03:11.210Z\",\r\n \"EndTime\": \"2026-06-10T14:03:13.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9012345\",\r\n \"Info\": \"Load Python Script: One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\",\r\n \"CreationTime\": \"2026-06-10T14:03:10.800Z\",\r\n \"ReleaseName\": \"PyDataTransform\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012332,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T14:03:13.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123\",\r\n \"Id\": 712440187\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90123", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T14:03:11.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-10T14:03:12.510Z\",\r\n \"Message\": \"[Python Scope] Initializing Python engine (Target: x64, Path: C:\\\\Python311, Version: Auto)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T14:03:13.560Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T14:03:13.600Z\",\r\n \"Message\": \"One or more errors occurred. ---> System.Exception: Error initializing Python engine 64 bit\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T14:03:13.620Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\",\r\n \"Name\": \"Data Pipelines\",\r\n \"Path\": \"Data Pipelines\",\r\n \"Description\": \"Production folder for data-transformation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/task.yaml index e16d9bebbe..39bcdde3d6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-engine-init/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/README.md new file mode 100644 index 0000000000..b258778670 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/README.md @@ -0,0 +1,46 @@ +# Load Python Script Hang - stdout Flood at Module Load (L3) + +This scenario reproduces a `Load Python Script` hang. `scripts/aggregate.py` has +a module-level loop that prints every row to stdout at import; because Load +Python Script executes the module body, the stdout flood saturates the engine +pipe and the load never completes within the 30s `Timeout`, so the scope faults +with a `TimeoutException`. + +## What this scenario uncovers + +**Root Cause:** A module-level `for _i in range(5_000_000): ... print(...)` loop +runs at import. The per-row stdout floods the engine's output pipe and the heavy +loop never finishes within `Timeout`, hanging the scope. + +This maps to: +`references/activity-packages/python-activities/playbooks/load-script-failures.md` +sub-case **L3** (hang or oversized data). + +The engine initializes fine and there is no syntax error or `ModuleNotFoundError` +- this is explicitly **not** the L1 engine-init or L2 script-error cases. The +user is framed as **off-host**, so the correct agent behavior is to tie the hang +to load-time stdout / heavy module-level work and recommend removing it (and/or +raising `Timeout`) - not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Python Scope` (`Timeout=30000`) -> `Load Python Script` -> `Invoke Python Method` and `scripts/aggregate.py` (module-level print-flood loop) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `load-script-failures.md` (sub-case L3) +- Agent identified a hang from load-time stdout flood / heavy module-level work + exceeding `Timeout` (not engine init, not a missing module) and recommended + removing/reducing the load-time stdout (move work into a function) and/or + raising `Timeout`, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/README.md deleted file mode 100644 index 8519345d00..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Load Python Script Hang - stdout Flood at Module Load (L3) - -This scenario reproduces a `Load Python Script` hang. `scripts/aggregate.py` has -a module-level loop that prints every row to stdout at import; because Load -Python Script executes the module body, the stdout flood saturates the engine -pipe and the load never completes within the 30s `Timeout`, so the scope faults -with a `TimeoutException`. - -## What this scenario uncovers - -**Root Cause:** A module-level `for _i in range(5_000_000): ... print(...)` loop -runs at import. The per-row stdout floods the engine's output pipe and the heavy -loop never finishes within `Timeout`, hanging the scope. - -This maps to: -`references/activity-packages/python-activities/playbooks/load-script-failures.md` -sub-case **L3** (hang or oversized data). - -The engine initializes fine and there is no syntax error or `ModuleNotFoundError` -- this is explicitly **not** the L1 engine-init or L2 script-error cases. The -user is framed as **off-host**, so the correct agent behavior is to tie the hang -to load-time stdout / heavy module-level work and recommend removing it (and/or -raising `Timeout`) - not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Python Scope` (`Timeout=30000`) -> `Load Python Script` -> `Invoke Python Method` and `scripts/aggregate.py` (module-level print-flood loop) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `load-script-failures.md` (sub-case L3) -- Agent identified a hang from load-time stdout flood / heavy module-level work - exceeding `Timeout` (not engine init, not a missing module) and recommended - removing/reducing the load-time stdout (move work into a function) and/or - raising `Timeout`, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/manifest.json deleted file mode 100644 index b4741e4666..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"aa02b2c3-d4e5-4f60-8a02-000000000002\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa02b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key aa02b2c3-d4e5-4f60-8a02-000000000002 --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa02b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"bb02c3d4-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-get-bb02c3d4-output-json.json" - }, - { - "match": "or jobs get bb02c3d4-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-get-bb02c3d4-output-json.json" - }, - { - "match": "or jobs logs \"bb02c3d4-e5f6-4071-9b02-000000000002\" --level Error --output json", - "file": "or-jobs-logs-bb02c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs bb02c3d4-e5f6-4071-9b02-000000000002 --level Error --output json", - "file": "or-jobs-logs-bb02c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs \"bb02c3d4-e5f6-4071-9b02-000000000002\" --output json", - "file": "or-jobs-logs-bb02c3d4-output-json.json" - }, - { - "match": "or jobs logs bb02c3d4-e5f6-4071-9b02-000000000002 --output json", - "file": "or-jobs-logs-bb02c3d4-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index bc7a13fb20..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "aa02b2c3-d4e5-4f60-8a02-000000000002", - "Name": "Batch Analytics", - "Path": "Batch Analytics", - "Description": "Production folder for batch aggregation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-get-bb02c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-get-bb02c3d4-output-json.json deleted file mode 100644 index dc55ec52d6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-get-bb02c3d4-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb02c3d4-e5f6-4071-9b02-000000000002", - "StartTime": "2026-06-11T10:15:02.110Z", - "EndTime": "2026-06-11T10:15:33.540Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc02d4e5-f6a7-4182-9c02-000000000002", - "Info": "Load Python Script: One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms). The Python standard output stream may be flooded.\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred.", - "CreationTime": "2026-06-11T10:15:01.700Z", - "ReleaseName": "PyBatchAggregate", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012402, - "OrganizationUnitId": 7791618, - "FolderKey": "aa02b2c3-d4e5-4f60-8a02-000000000002", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T10:15:33.540Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "bb02c3d4-e5f6-4071-9b02-000000000002", - "Id": 712450202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-list-folder-key-aa02b2c3-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-list-folder-key-aa02b2c3-state-faulted-output-json.json deleted file mode 100644 index 1bcdcce61a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-list-folder-key-aa02b2c3-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "bb02c3d4-e5f6-4071-9b02-000000000002", - "State": "Faulted", - "ProcessName": "PyBatchAggregate", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-11T10:15:02.110Z", - "EndTime": "2026-06-11T10:15:33.540Z", - "CreationTime": "2026-06-11T10:15:01.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-logs-bb02c3d4-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-logs-bb02c3d4-level-error-output-json.json deleted file mode 100644 index fca63b6c6b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-logs-bb02c3d4-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T10:15:33.380Z", - "Message": "[Load Python Script] One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms). The Python standard output stream may be flooded." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T10:15:33.470Z", - "Message": "One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms).\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-logs-bb02c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-logs-bb02c3d4-output-json.json deleted file mode 100644 index f10e770452..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/m/r/or-jobs-logs-bb02c3d4-output-json.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-11T10:15:02.140Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T10:15:03.060Z", - "Message": "[Python Scope] Initializing Python engine (Target: x64, Path: C:\\Python39, Version: Python_39)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T10:15:03.520Z", - "Message": "[Python Scope] Python engine initialized." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T10:15:03.780Z", - "Message": "[Load Python Script] Loading scripts\\aggregate.py" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T10:15:08.940Z", - "Message": "[Load Python Script] stdout: warmup row 612344 => 375165... (very large volume of per-row output continuing)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T10:15:20.110Z", - "Message": "[Load Python Script] stdout: warmup row 2487051 => 6185... (engine output buffer saturated)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T10:15:33.380Z", - "Message": "[Load Python Script] One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms). The Python standard output stream may be flooded." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T10:15:33.470Z", - "Message": "One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms).\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T10:15:33.520Z", - "Message": "Execution ended in: 00:00:31" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/uip-fixture.json new file mode 100644 index 0000000000..7c6a7c7da6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "aa02b2c3-d4e5-4f60-8a02-000000000002", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb02c3d4-e5f6-4071-9b02-000000000002\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyBatchAggregate\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T10:15:02.110Z\",\r\n \"EndTime\": \"2026-06-11T10:15:33.540Z\",\r\n \"CreationTime\": \"2026-06-11T10:15:01.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb02c3d4-e5f6-4071-9b02-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T10:15:33.380Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms). The Python standard output stream may be flooded.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T10:15:33.470Z\",\r\n \"Message\": \"One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms).\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb02c3d4-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb02c3d4-e5f6-4071-9b02-000000000002\",\r\n \"StartTime\": \"2026-06-11T10:15:02.110Z\",\r\n \"EndTime\": \"2026-06-11T10:15:33.540Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc02d4e5-f6a7-4182-9c02-000000000002\",\r\n \"Info\": \"Load Python Script: One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms). The Python standard output stream may be flooded.\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred.\",\r\n \"CreationTime\": \"2026-06-11T10:15:01.700Z\",\r\n \"ReleaseName\": \"PyBatchAggregate\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012402,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"aa02b2c3-d4e5-4f60-8a02-000000000002\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T10:15:33.540Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bb02c3d4-e5f6-4071-9b02-000000000002\",\r\n \"Id\": 712450202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb02c3d4-e5f6-4071-9b02-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T10:15:02.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T10:15:03.060Z\",\r\n \"Message\": \"[Python Scope] Initializing Python engine (Target: x64, Path: C:\\\\Python39, Version: Python_39)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T10:15:03.520Z\",\r\n \"Message\": \"[Python Scope] Python engine initialized.\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T10:15:03.780Z\",\r\n \"Message\": \"[Load Python Script] Loading scripts\\\\aggregate.py\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T10:15:08.940Z\",\r\n \"Message\": \"[Load Python Script] stdout: warmup row 612344 => 375165... (very large volume of per-row output continuing)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T10:15:20.110Z\",\r\n \"Message\": \"[Load Python Script] stdout: warmup row 2487051 => 6185... (engine output buffer saturated)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T10:15:33.380Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms). The Python standard output stream may be flooded.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T10:15:33.470Z\",\r\n \"Message\": \"One or more errors occurred. ---> System.TimeoutException: The Python script did not complete within the configured Timeout (30000 ms).\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T10:15:33.520Z\",\r\n \"Message\": \"Execution ended in: 00:00:31\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa02b2c3-d4e5-4f60-8a02-000000000002\",\r\n \"Name\": \"Batch Analytics\",\r\n \"Path\": \"Batch Analytics\",\r\n \"Description\": \"Production folder for batch aggregation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/task.yaml index 03e11b9fd6..b36e2411f9 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-load-hang/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/README.md new file mode 100644 index 0000000000..6041bc4ffc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/README.md @@ -0,0 +1,47 @@ +# Load Python Script Failure - Unresolved Local Import / WorkingFolder (L2c) + +This scenario reproduces a `Load Python Script` failure where a **sibling local +import** is not found. `scripts/ledger_sync.py` does `import ledger_utils` (a +local module that exists in the same `scripts/` folder), but the `Python Scope` +has no `WorkingFolder` set, so the import resolves against the robot's package +directory instead of `scripts/` and faults at load with `ModuleNotFoundError: No +module named 'ledger_utils'`. + +## What this scenario uncovers + +**Root Cause:** The local sibling module `ledger_utils.py` is present in the +project, but the scope's `WorkingFolder` is unset, so the interpreter's import +path does not include `scripts/`. The sibling import fails at load. + +This maps to: +`references/activity-packages/python-activities/playbooks/load-script-failures.md` +sub-case **L2c** (unresolved local import). + +The discriminator vs. **L2a**: `ledger_utils` is the user's **own file in the +repo**, not a third-party package — so the fix is `WorkingFolder`, **not** `pip +install`. The user is framed as **off-host**; the correct agent behavior is to +recognize the local import and recommend setting `WorkingFolder` - not to attempt +host commands or install a package. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Python Scope` (no `WorkingFolder`) -> `Load Python Script` -> `Invoke Python Method`, plus `scripts/ledger_sync.py` (imports `ledger_utils`) and the sibling `scripts/ledger_utils.py` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `load-script-failures.md` (sub-case L2c) +- Agent recognized a local sibling-module import failing because `WorkingFolder` + is not the script's folder (not a missing third-party package) and recommended + setting `WorkingFolder` to the script folder (not `pip install`), without + fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/README.md deleted file mode 100644 index 05deec363a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Load Python Script Failure - Unresolved Local Import / WorkingFolder (L2c) - -This scenario reproduces a `Load Python Script` failure where a **sibling local -import** is not found. `scripts/ledger_sync.py` does `import ledger_utils` (a -local module that exists in the same `scripts/` folder), but the `Python Scope` -has no `WorkingFolder` set, so the import resolves against the robot's package -directory instead of `scripts/` and faults at load with `ModuleNotFoundError: No -module named 'ledger_utils'`. - -## What this scenario uncovers - -**Root Cause:** The local sibling module `ledger_utils.py` is present in the -project, but the scope's `WorkingFolder` is unset, so the interpreter's import -path does not include `scripts/`. The sibling import fails at load. - -This maps to: -`references/activity-packages/python-activities/playbooks/load-script-failures.md` -sub-case **L2c** (unresolved local import). - -The discriminator vs. **L2a**: `ledger_utils` is the user's **own file in the -repo**, not a third-party package — so the fix is `WorkingFolder`, **not** `pip -install`. The user is framed as **off-host**; the correct agent behavior is to -recognize the local import and recommend setting `WorkingFolder` - not to attempt -host commands or install a package. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Python Scope` (no `WorkingFolder`) -> `Load Python Script` -> `Invoke Python Method`, plus `scripts/ledger_sync.py` (imports `ledger_utils`) and the sibling `scripts/ledger_utils.py` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `load-script-failures.md` (sub-case L2c) -- Agent recognized a local sibling-module import failing because `WorkingFolder` - is not the script's folder (not a missing third-party package) and recommended - setting `WorkingFolder` to the script folder (not `pip install`), without - fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/manifest.json deleted file mode 100644 index 5f66cfffdc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"aa03b2c3-d4e5-4f60-8a03-000000000003\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa03b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key aa03b2c3-d4e5-4f60-8a03-000000000003 --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa03b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"bb03c3d4-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-get-bb03c3d4-output-json.json" - }, - { - "match": "or jobs get bb03c3d4-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-get-bb03c3d4-output-json.json" - }, - { - "match": "or jobs logs \"bb03c3d4-e5f6-4071-9b03-000000000003\" --level Error --output json", - "file": "or-jobs-logs-bb03c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs bb03c3d4-e5f6-4071-9b03-000000000003 --level Error --output json", - "file": "or-jobs-logs-bb03c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs \"bb03c3d4-e5f6-4071-9b03-000000000003\" --output json", - "file": "or-jobs-logs-bb03c3d4-output-json.json" - }, - { - "match": "or jobs logs bb03c3d4-e5f6-4071-9b03-000000000003 --output json", - "file": "or-jobs-logs-bb03c3d4-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index da6374f1ff..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "aa03b2c3-d4e5-4f60-8a03-000000000003", - "Name": "Finance Sync", - "Path": "Finance Sync", - "Description": "Production folder for ledger synchronization processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-get-bb03c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-get-bb03c3d4-output-json.json deleted file mode 100644 index aebc627a31..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-get-bb03c3d4-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb03c3d4-e5f6-4071-9b03-000000000003", - "StartTime": "2026-06-11T13:22:48.210Z", - "EndTime": "2026-06-11T13:22:50.470Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc03d4e5-f6a7-4182-9c03-000000000003", - "Info": "Load Python Script: One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ModuleNotFoundError: No module named 'ledger_utils'\r\n File \"ledger_sync.py\", line 7, in \r\n import ledger_utils\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred.", - "CreationTime": "2026-06-11T13:22:47.800Z", - "ReleaseName": "PyLedgerSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012403, - "OrganizationUnitId": 7791618, - "FolderKey": "aa03b2c3-d4e5-4f60-8a03-000000000003", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T13:22:50.470Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "bb03c3d4-e5f6-4071-9b03-000000000003", - "Id": 712450303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-list-folder-key-aa03b2c3-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-list-folder-key-aa03b2c3-state-faulted-output-json.json deleted file mode 100644 index 943c82ea25..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-list-folder-key-aa03b2c3-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "bb03c3d4-e5f6-4071-9b03-000000000003", - "State": "Faulted", - "ProcessName": "PyLedgerSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-11T13:22:48.210Z", - "EndTime": "2026-06-11T13:22:50.470Z", - "CreationTime": "2026-06-11T13:22:47.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-logs-bb03c3d4-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-logs-bb03c3d4-level-error-output-json.json deleted file mode 100644 index 4258e6a1b0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-logs-bb03c3d4-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T13:22:50.300Z", - "Message": "[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ModuleNotFoundError: No module named 'ledger_utils' (ledger_sync.py, line 7)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T13:22:50.380Z", - "Message": "One or more errors occurred. ---> Python.Runtime.PythonException: ModuleNotFoundError: No module named 'ledger_utils'\r\n File \"ledger_sync.py\", line 7, in \r\n import ledger_utils\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-logs-bb03c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-logs-bb03c3d4-output-json.json deleted file mode 100644 index 8b099c0716..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/m/r/or-jobs-logs-bb03c3d4-output-json.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-11T13:22:48.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T13:22:49.160Z", - "Message": "[Python Scope] Initializing Python engine (Target: x64, Path: C:\\Python39, Version: Python_39)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T13:22:49.610Z", - "Message": "[Python Scope] Python engine initialized. WorkingFolder resolved to the package directory (no WorkingFolder set)." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T13:22:49.880Z", - "Message": "[Load Python Script] Loading scripts\\ledger_sync.py" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T13:22:50.300Z", - "Message": "[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ModuleNotFoundError: No module named 'ledger_utils' (ledger_sync.py, line 7)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T13:22:50.380Z", - "Message": "One or more errors occurred. ---> Python.Runtime.PythonException: ModuleNotFoundError: No module named 'ledger_utils'\r\n File \"ledger_sync.py\", line 7, in \r\n import ledger_utils\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T13:22:50.450Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/uip-fixture.json new file mode 100644 index 0000000000..c5b59dcd92 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "aa03b2c3-d4e5-4f60-8a03-000000000003", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb03c3d4-e5f6-4071-9b03-000000000003\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyLedgerSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T13:22:48.210Z\",\r\n \"EndTime\": \"2026-06-11T13:22:50.470Z\",\r\n \"CreationTime\": \"2026-06-11T13:22:47.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb03c3d4-e5f6-4071-9b03-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T13:22:50.300Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ModuleNotFoundError: No module named 'ledger_utils' (ledger_sync.py, line 7)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T13:22:50.380Z\",\r\n \"Message\": \"One or more errors occurred. ---> Python.Runtime.PythonException: ModuleNotFoundError: No module named 'ledger_utils'\\r\\n File \\\"ledger_sync.py\\\", line 7, in \\r\\n import ledger_utils\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb03c3d4-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb03c3d4-e5f6-4071-9b03-000000000003\",\r\n \"StartTime\": \"2026-06-11T13:22:48.210Z\",\r\n \"EndTime\": \"2026-06-11T13:22:50.470Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc03d4e5-f6a7-4182-9c03-000000000003\",\r\n \"Info\": \"Load Python Script: One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ModuleNotFoundError: No module named 'ledger_utils'\\r\\n File \\\"ledger_sync.py\\\", line 7, in \\r\\n import ledger_utils\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred.\",\r\n \"CreationTime\": \"2026-06-11T13:22:47.800Z\",\r\n \"ReleaseName\": \"PyLedgerSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012403,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"aa03b2c3-d4e5-4f60-8a03-000000000003\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T13:22:50.470Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bb03c3d4-e5f6-4071-9b03-000000000003\",\r\n \"Id\": 712450303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb03c3d4-e5f6-4071-9b03-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T13:22:48.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T13:22:49.160Z\",\r\n \"Message\": \"[Python Scope] Initializing Python engine (Target: x64, Path: C:\\\\Python39, Version: Python_39)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T13:22:49.610Z\",\r\n \"Message\": \"[Python Scope] Python engine initialized. WorkingFolder resolved to the package directory (no WorkingFolder set).\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T13:22:49.880Z\",\r\n \"Message\": \"[Load Python Script] Loading scripts\\\\ledger_sync.py\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T13:22:50.300Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ModuleNotFoundError: No module named 'ledger_utils' (ledger_sync.py, line 7)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T13:22:50.380Z\",\r\n \"Message\": \"One or more errors occurred. ---> Python.Runtime.PythonException: ModuleNotFoundError: No module named 'ledger_utils'\\r\\n File \\\"ledger_sync.py\\\", line 7, in \\r\\n import ledger_utils\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T13:22:50.450Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa03b2c3-d4e5-4f60-8a03-000000000003\",\r\n \"Name\": \"Finance Sync\",\r\n \"Path\": \"Finance Sync\",\r\n \"Description\": \"Production folder for ledger synchronization processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/task.yaml index 95884b698b..bf60511a4f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-local-import/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/README.md new file mode 100644 index 0000000000..5ae5a253e4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/README.md @@ -0,0 +1,47 @@ +# Load Python Script Failure - Top-Level Exception at Module Load (L2b) + +This scenario reproduces a `Load Python Script` failure caused by **module-level +(top-level) code that raises at import time**. `Load Python Script` executes the +module body to bind its functions, so a statement at module scope that throws +aborts the load with `Error loading the python script` / `One or more errors +occurred` before any function is invoked. + +## What this scenario uncovers + +**Root Cause:** `scripts/order_import.py` has a module-level statement +`DEFAULT_DISCOUNT = 100 / 0` that raises `ZeroDivisionError` when the module is +imported. Load Python Script runs the module body at load, so the top-level line +faults the activity before `import_orders` is ever called. + +This maps to: +`references/activity-packages/python-activities/playbooks/load-script-failures.md` +sub-case **L2b** (script load / compile / top-level exception). + +The engine initializes fine (logs show "Python engine initialized") and there is +no `ModuleNotFoundError`, so this is explicitly **not** the L1 engine-init case +or the L2a missing-package case. The user is framed as **off-host**, so the +correct agent behavior is to tie the load error to top-level code and recommend +guarding it - not to attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Python Scope` -> `Load Python Script` -> `Invoke Python Method` and `scripts/order_import.py` (module-level divide-by-zero) | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `load-script-failures.md` (sub-case L2b) +- Agent identified a top-level / module-scope exception at load as the cause + (not engine init, not ModuleNotFoundError) and recommended moving the code + into a function or behind `if __name__ == "__main__":`, without fabricating + host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/README.md deleted file mode 100644 index 585fd4693d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Load Python Script Failure - Top-Level Exception at Module Load (L2b) - -This scenario reproduces a `Load Python Script` failure caused by **module-level -(top-level) code that raises at import time**. `Load Python Script` executes the -module body to bind its functions, so a statement at module scope that throws -aborts the load with `Error loading the python script` / `One or more errors -occurred` before any function is invoked. - -## What this scenario uncovers - -**Root Cause:** `scripts/order_import.py` has a module-level statement -`DEFAULT_DISCOUNT = 100 / 0` that raises `ZeroDivisionError` when the module is -imported. Load Python Script runs the module body at load, so the top-level line -faults the activity before `import_orders` is ever called. - -This maps to: -`references/activity-packages/python-activities/playbooks/load-script-failures.md` -sub-case **L2b** (script load / compile / top-level exception). - -The engine initializes fine (logs show "Python engine initialized") and there is -no `ModuleNotFoundError`, so this is explicitly **not** the L1 engine-init case -or the L2a missing-package case. The user is framed as **off-host**, so the -correct agent behavior is to tie the load error to top-level code and recommend -guarding it - not to attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Python Scope` -> `Load Python Script` -> `Invoke Python Method` and `scripts/order_import.py` (module-level divide-by-zero) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `load-script-failures.md` (sub-case L2b) -- Agent identified a top-level / module-scope exception at load as the cause - (not engine init, not ModuleNotFoundError) and recommended moving the code - into a function or behind `if __name__ == "__main__":`, without fabricating - host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/manifest.json deleted file mode 100644 index ed3ee016a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"aa01b2c3-d4e5-4f60-8a01-000000000001\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa01b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key aa01b2c3-d4e5-4f60-8a01-000000000001 --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa01b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"bb01c3d4-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-get-bb01c3d4-output-json.json" - }, - { - "match": "or jobs get bb01c3d4-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-get-bb01c3d4-output-json.json" - }, - { - "match": "or jobs logs \"bb01c3d4-e5f6-4071-9b01-000000000001\" --level Error --output json", - "file": "or-jobs-logs-bb01c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs bb01c3d4-e5f6-4071-9b01-000000000001 --level Error --output json", - "file": "or-jobs-logs-bb01c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs \"bb01c3d4-e5f6-4071-9b01-000000000001\" --output json", - "file": "or-jobs-logs-bb01c3d4-output-json.json" - }, - { - "match": "or jobs logs bb01c3d4-e5f6-4071-9b01-000000000001 --output json", - "file": "or-jobs-logs-bb01c3d4-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 9a4ce6e3c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "aa01b2c3-d4e5-4f60-8a01-000000000001", - "Name": "Order Processing", - "Path": "Order Processing", - "Description": "Production folder for order-import processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-get-bb01c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-get-bb01c3d4-output-json.json deleted file mode 100644 index 9d8e710738..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-get-bb01c3d4-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb01c3d4-e5f6-4071-9b01-000000000001", - "StartTime": "2026-06-11T08:41:05.210Z", - "EndTime": "2026-06-11T08:41:07.480Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc01d4e5-f6a7-4182-9c01-000000000001", - "Info": "Load Python Script: One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ZeroDivisionError: division by zero\r\n File \"order_import.py\", line 21, in \r\n DEFAULT_DISCOUNT = 100 / 0\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred.", - "CreationTime": "2026-06-11T08:41:04.800Z", - "ReleaseName": "PyOrderImport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012401, - "OrganizationUnitId": 7791618, - "FolderKey": "aa01b2c3-d4e5-4f60-8a01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T08:41:07.480Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "bb01c3d4-e5f6-4071-9b01-000000000001", - "Id": 712450101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-list-folder-key-aa01b2c3-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-list-folder-key-aa01b2c3-state-faulted-output-json.json deleted file mode 100644 index 961ab8f942..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-list-folder-key-aa01b2c3-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "bb01c3d4-e5f6-4071-9b01-000000000001", - "State": "Faulted", - "ProcessName": "PyOrderImport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-11T08:41:05.210Z", - "EndTime": "2026-06-11T08:41:07.480Z", - "CreationTime": "2026-06-11T08:41:04.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-logs-bb01c3d4-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-logs-bb01c3d4-level-error-output-json.json deleted file mode 100644 index b67f3cda09..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-logs-bb01c3d4-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:41:07.360Z", - "Message": "[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ZeroDivisionError: division by zero (order_import.py, line 21)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:41:07.420Z", - "Message": "One or more errors occurred. ---> Python.Runtime.PythonException: ZeroDivisionError: division by zero\r\n File \"order_import.py\", line 21, in \r\n DEFAULT_DISCOUNT = 100 / 0\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-logs-bb01c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-logs-bb01c3d4-output-json.json deleted file mode 100644 index c4343a1deb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/m/r/or-jobs-logs-bb01c3d4-output-json.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-11T08:41:05.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T08:41:06.180Z", - "Message": "[Python Scope] Initializing Python engine (Target: x64, Path: C:\\Python39, Version: Python_39)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T08:41:06.640Z", - "Message": "[Python Scope] Python engine initialized." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T08:41:06.900Z", - "Message": "[Load Python Script] Loading scripts\\order_import.py" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:41:07.360Z", - "Message": "[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ZeroDivisionError: division by zero (order_import.py, line 21)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T08:41:07.420Z", - "Message": "One or more errors occurred. ---> Python.Runtime.PythonException: ZeroDivisionError: division by zero\r\n File \"order_import.py\", line 21, in \r\n DEFAULT_DISCOUNT = 100 / 0\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T08:41:07.460Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/uip-fixture.json new file mode 100644 index 0000000000..b7f2f7dce8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "aa01b2c3-d4e5-4f60-8a01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb01c3d4-e5f6-4071-9b01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyOrderImport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T08:41:05.210Z\",\r\n \"EndTime\": \"2026-06-11T08:41:07.480Z\",\r\n \"CreationTime\": \"2026-06-11T08:41:04.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb01c3d4-e5f6-4071-9b01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:41:07.360Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ZeroDivisionError: division by zero (order_import.py, line 21)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:41:07.420Z\",\r\n \"Message\": \"One or more errors occurred. ---> Python.Runtime.PythonException: ZeroDivisionError: division by zero\\r\\n File \\\"order_import.py\\\", line 21, in \\r\\n DEFAULT_DISCOUNT = 100 / 0\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb01c3d4-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb01c3d4-e5f6-4071-9b01-000000000001\",\r\n \"StartTime\": \"2026-06-11T08:41:05.210Z\",\r\n \"EndTime\": \"2026-06-11T08:41:07.480Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc01d4e5-f6a7-4182-9c01-000000000001\",\r\n \"Info\": \"Load Python Script: One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ZeroDivisionError: division by zero\\r\\n File \\\"order_import.py\\\", line 21, in \\r\\n DEFAULT_DISCOUNT = 100 / 0\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred.\",\r\n \"CreationTime\": \"2026-06-11T08:41:04.800Z\",\r\n \"ReleaseName\": \"PyOrderImport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012401,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"aa01b2c3-d4e5-4f60-8a01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T08:41:07.480Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bb01c3d4-e5f6-4071-9b01-000000000001\",\r\n \"Id\": 712450101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb01c3d4-e5f6-4071-9b01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T08:41:05.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T08:41:06.180Z\",\r\n \"Message\": \"[Python Scope] Initializing Python engine (Target: x64, Path: C:\\\\Python39, Version: Python_39)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T08:41:06.640Z\",\r\n \"Message\": \"[Python Scope] Python engine initialized.\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T08:41:06.900Z\",\r\n \"Message\": \"[Load Python Script] Loading scripts\\\\order_import.py\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:41:07.360Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> Python.Runtime.PythonException: Error loading the python script: ZeroDivisionError: division by zero (order_import.py, line 21)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T08:41:07.420Z\",\r\n \"Message\": \"One or more errors occurred. ---> Python.Runtime.PythonException: ZeroDivisionError: division by zero\\r\\n File \\\"order_import.py\\\", line 21, in \\r\\n DEFAULT_DISCOUNT = 100 / 0\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T08:41:07.460Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa01b2c3-d4e5-4f60-8a01-000000000001\",\r\n \"Name\": \"Order Processing\",\r\n \"Path\": \"Order Processing\",\r\n \"Description\": \"Production folder for order-import processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/task.yaml index 292b57ce94..10e4af6c8a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-toplevel-exception/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/README.md new file mode 100644 index 0000000000..7c67d67975 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/README.md @@ -0,0 +1,52 @@ +# Load Python Script Engine-Init Failure - Python Version Unsupported by the Package (L1d) + +This scenario reproduces a `Load Python Script` engine-initialization failure +caused by the **installed Python being newer than the package supports**. The +project pins `UiPath.Python.Activities` 1.8.1 (which supports up to Python 3.12), +but the robot now has Python 3.13 (`Version=Auto` detects 3.13), so the scope +cannot initialize the engine and faults with `Error initializing the Python +engine. The detected Python version 3.13 is not supported by +UiPath.Python.Activities 1.8.1`. + +## What this scenario uncovers + +**Root Cause:** A version-support gap between the pinned package (1.8.1, max +Python 3.12) and the interpreter on the robot (3.13). The pack cannot stand up an +engine for an unsupported Python version, so the failure is at engine init, +before the script loads. + +This maps to: +`references/activity-packages/python-activities/playbooks/load-script-failures.md` +sub-case **L1d** (engine init - Version unsupported by the package). + +The `Library path` is set correctly (`python313.dll`) and `Target` matches, so +this is explicitly **not** L1c (library path) or L1b (bitness); and the script +never loads, so it is **not** a `ModuleNotFoundError` or top-level error. The +user is framed as **off-host**; the correct agent behavior is to recommend +upgrading the package (or pinning a supported Python version) - not to attempt +host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project pinning `UiPath.Python.Activities` 1.8.1, a `Python Scope` (`Version=Auto`, Library path set) -> `Load Python Script` -> `Invoke Python Method`, and a valid `scripts/forecast.py` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. The Python-version-to-package support map +> (3.12 -> 1.8.1, 3.13 -> 1.10.0) is taken from `load-script-failures.md`. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `load-script-failures.md` (sub-case L1d) +- Agent identified an engine-init failure from the installed/detected Python + version being unsupported by the pinned package (not Library path, not bitness, + not a missing module) and recommended upgrading the package to a version + supporting Python 3.13 (1.10.0) or pinning `Version` to a supported + interpreter, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/README.md deleted file mode 100644 index 002522b888..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Load Python Script Engine-Init Failure - Python Version Unsupported by the Package (L1d) - -This scenario reproduces a `Load Python Script` engine-initialization failure -caused by the **installed Python being newer than the package supports**. The -project pins `UiPath.Python.Activities` 1.8.1 (which supports up to Python 3.12), -but the robot now has Python 3.13 (`Version=Auto` detects 3.13), so the scope -cannot initialize the engine and faults with `Error initializing the Python -engine. The detected Python version 3.13 is not supported by -UiPath.Python.Activities 1.8.1`. - -## What this scenario uncovers - -**Root Cause:** A version-support gap between the pinned package (1.8.1, max -Python 3.12) and the interpreter on the robot (3.13). The pack cannot stand up an -engine for an unsupported Python version, so the failure is at engine init, -before the script loads. - -This maps to: -`references/activity-packages/python-activities/playbooks/load-script-failures.md` -sub-case **L1d** (engine init - Version unsupported by the package). - -The `Library path` is set correctly (`python313.dll`) and `Target` matches, so -this is explicitly **not** L1c (library path) or L1b (bitness); and the script -never loads, so it is **not** a `ModuleNotFoundError` or top-level error. The -user is framed as **off-host**; the correct agent behavior is to recommend -upgrading the package (or pinning a supported Python version) - not to attempt -host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project pinning `UiPath.Python.Activities` 1.8.1, a `Python Scope` (`Version=Auto`, Library path set) -> `Load Python Script` -> `Invoke Python Method`, and a valid `scripts/forecast.py` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. The Python-version-to-package support map -> (3.12 -> 1.8.1, 3.13 -> 1.10.0) is taken from `load-script-failures.md`. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `load-script-failures.md` (sub-case L1d) -- Agent identified an engine-init failure from the installed/detected Python - version being unsupported by the pinned package (not Library path, not bitness, - not a missing module) and recommended upgrading the package to a version - supporting Python 3.13 (1.10.0) or pinning `Version` to a supported - interpreter, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/manifest.json deleted file mode 100644 index 2a20c635b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"aa04b2c3-d4e5-4f60-8a04-000000000004\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa04b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key aa04b2c3-d4e5-4f60-8a04-000000000004 --state Faulted --output json", - "file": "or-jobs-list-folder-key-aa04b2c3-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"bb04c3d4-e5f6-4071-9b04-000000000004\" --output json", - "file": "or-jobs-get-bb04c3d4-output-json.json" - }, - { - "match": "or jobs get bb04c3d4-e5f6-4071-9b04-000000000004 --output json", - "file": "or-jobs-get-bb04c3d4-output-json.json" - }, - { - "match": "or jobs logs \"bb04c3d4-e5f6-4071-9b04-000000000004\" --level Error --output json", - "file": "or-jobs-logs-bb04c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs bb04c3d4-e5f6-4071-9b04-000000000004 --level Error --output json", - "file": "or-jobs-logs-bb04c3d4-level-error-output-json.json" - }, - { - "match": "or jobs logs \"bb04c3d4-e5f6-4071-9b04-000000000004\" --output json", - "file": "or-jobs-logs-bb04c3d4-output-json.json" - }, - { - "match": "or jobs logs bb04c3d4-e5f6-4071-9b04-000000000004 --output json", - "file": "or-jobs-logs-bb04c3d4-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 3d4e4230ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "aa04b2c3-d4e5-4f60-8a04-000000000004", - "Name": "Forecasting", - "Path": "Forecasting", - "Description": "Production folder for forecasting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-get-bb04c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-get-bb04c3d4-output-json.json deleted file mode 100644 index 7df500319c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-get-bb04c3d4-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb04c3d4-e5f6-4071-9b04-000000000004", - "StartTime": "2026-06-11T15:48:19.210Z", - "EndTime": "2026-06-11T15:48:21.360Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cc04d4e5-f6a7-4182-9c04-000000000004", - "Info": "Load Python Script: One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred.", - "CreationTime": "2026-06-11T15:48:18.800Z", - "ReleaseName": "PyForecast", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012404, - "OrganizationUnitId": 7791618, - "FolderKey": "aa04b2c3-d4e5-4f60-8a04-000000000004", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T15:48:21.360Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "bb04c3d4-e5f6-4071-9b04-000000000004", - "Id": 712450404 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-list-folder-key-aa04b2c3-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-list-folder-key-aa04b2c3-state-faulted-output-json.json deleted file mode 100644 index 635460a473..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-list-folder-key-aa04b2c3-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "bb04c3d4-e5f6-4071-9b04-000000000004", - "State": "Faulted", - "ProcessName": "PyForecast", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-11T15:48:19.210Z", - "EndTime": "2026-06-11T15:48:21.360Z", - "CreationTime": "2026-06-11T15:48:18.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-logs-bb04c3d4-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-logs-bb04c3d4-level-error-output-json.json deleted file mode 100644 index 32fca11b8a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-logs-bb04c3d4-level-error-output-json.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-11T15:48:21.180Z", - "Message": "[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T15:48:21.260Z", - "Message": "One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-logs-bb04c3d4-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-logs-bb04c3d4-output-json.json deleted file mode 100644 index 0fcfb94acd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/m/r/or-jobs-logs-bb04c3d4-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-11T15:48:19.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T15:48:20.090Z", - "Message": "[Python Scope] Initializing Python engine (Target: x64, Path: C:\\Python313, Version: Auto -> detected 3.13)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T15:48:21.180Z", - "Message": "[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-11T15:48:21.260Z", - "Message": "One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at LoadScript \"Load Python Script\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T15:48:21.330Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/uip-fixture.json new file mode 100644 index 0000000000..f2eb957079 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "aa04b2c3-d4e5-4f60-8a04-000000000004", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb04c3d4-e5f6-4071-9b04-000000000004\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyForecast\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T15:48:19.210Z\",\r\n \"EndTime\": \"2026-06-11T15:48:21.360Z\",\r\n \"CreationTime\": \"2026-06-11T15:48:18.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb04c3d4-e5f6-4071-9b04-000000000004", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T15:48:21.180Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T15:48:21.260Z\",\r\n \"Message\": \"One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bb04c3d4-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb04c3d4-e5f6-4071-9b04-000000000004\",\r\n \"StartTime\": \"2026-06-11T15:48:19.210Z\",\r\n \"EndTime\": \"2026-06-11T15:48:21.360Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cc04d4e5-f6a7-4182-9c04-000000000004\",\r\n \"Info\": \"Load Python Script: One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred.\",\r\n \"CreationTime\": \"2026-06-11T15:48:18.800Z\",\r\n \"ReleaseName\": \"PyForecast\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012404,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"aa04b2c3-d4e5-4f60-8a04-000000000004\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T15:48:21.360Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bb04c3d4-e5f6-4071-9b04-000000000004\",\r\n \"Id\": 712450404\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb04c3d4-e5f6-4071-9b04-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T15:48:19.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T15:48:20.090Z\",\r\n \"Message\": \"[Python Scope] Initializing Python engine (Target: x64, Path: C:\\\\Python313, Version: Auto -> detected 3.13)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T15:48:21.180Z\",\r\n \"Message\": \"[Load Python Script] One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-11T15:48:21.260Z\",\r\n \"Message\": \"One or more errors occurred. ---> System.Exception: Error initializing the Python engine. The detected Python version 3.13 is not supported by UiPath.Python.Activities 1.8.1.\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at LoadScript \\\"Load Python Script\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T15:48:21.330Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa04b2c3-d4e5-4f60-8a04-000000000004\",\r\n \"Name\": \"Forecasting\",\r\n \"Path\": \"Forecasting\",\r\n \"Description\": \"Production folder for forecasting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/task.yaml index c8d2090ecf..c2f07f4012 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-loadscript-version-unsupported/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/README.md new file mode 100644 index 0000000000..729ad9e4a3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/README.md @@ -0,0 +1,45 @@ +# Python Scope — Pipe is Broken (Missing pip Module in the Robot Interpreter) + +This scenario reproduces a `Python Scope` failure where `Invoke Python Method` +faults with `Pipe is broken`. The invoked script imports `pandas`, which is +installed in the developer's IDE interpreter but **not** in the Python 3.11 +interpreter the `Python Scope` resolves on the robot host. The job ends with: + +``` +UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: "Invoke Python Method: extract_total" +``` + +## What this scenario uncovers + +**Root Cause:** `UiPath.Python.Activities` runs Python out-of-process and +talks to it over an IPC pipe. `parse_invoice.py` does `import pandas`. The +robot's interpreter (`C:\Program Files\Python311`, the one `Python Scope` +`Path` resolves) does not have `pandas` installed, so the Python host raises +`ModuleNotFoundError` and exits before returning. UiPath only sees the host +die and reports `Pipe is broken` — the underlying Python error is hidden. +The script "runs fine in the developer's IDE" because the IDE uses a +different interpreter that has `pandas`. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/python-activities/playbooks/invoke-python-method-pipe-is-broken.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `PythonInvoiceParser` — `Python Scope` (`Path=C:\Program Files\Python311`, `Target=x64`, `Version=Python_311`) → `Load Python Script: parse_invoice.py` → `Invoke Python Method: extract_total`; `parse_invoice.py` imports `pandas` | + +The decisive evidence is **not** in the job log (the Python traceback is +swallowed): it is the `import pandas` line in `process/parse_invoice.py` +combined with the user's "runs fine in my IDE" framing. The test grades +whether the agent identifies that the out-of-process Python host died on a +missing third-party module in the robot's interpreter, and recommends +installing it into the scope's interpreter / reproducing the real traceback +standalone. + +> **Note on fixtures.** Synthetic. Job key, folder key, and host are +> placeholders representative of a real Robot host layout. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/2c90546e13.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/2c90546e13.json deleted file mode 100644 index f56491df0e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/2c90546e13.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-06-18T09:15:02.110Z", - "EndTime": "2026-06-18T09:15:08.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \"Invoke Python Method: extract_total\"\r\n at UiPath.Python.Service.PythonClient.InvokeMethod(PythonObject instance, String method, IList`1 args)\r\n at UiPath.Python.Activities.InvokeMethod.<>c__DisplayClass.b__0()\r\n at UiPath.Python.Activities.InvokeMethod.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-06-18T09:15:01.900Z", - "ReleaseName": "PythonInvoiceParser", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OrganizationUnitId": 6020001, - "FolderKey": "fa111111-2222-3333-4444-555566667777", - "Reference": "InvoiceRunA", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "LastModificationTime": "2026-06-18T09:15:08.640Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "RemoteException", - "ActivityName": "InvokeMethod_1", - "ActivityDisplayName": "Invoke Python Method: extract_total", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "Id": 680078101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/79723a4720.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/79723a4720.json deleted file mode 100644 index cedf7426c5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/79723a4720.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa111111-001", - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-06-18T09:15:02.180Z", - "Message": "Execution started.", - "ProcessName": "PythonInvoiceParser", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-002", - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-06-18T09:15:02.260Z", - "Message": "[PythonInvoiceParser] Invoking parse_invoice.extract_total via Python Scope", - "ProcessName": "PythonInvoiceParser", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-003", - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Trace", - "Timestamp": "2026-06-18T09:15:04.020Z", - "Message": "Python Scope — resolved interpreter at 'C:\\Program Files\\Python311' (Version=Python_311, Target=x64); host process started.", - "ProcessName": "PythonInvoiceParser", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-004", - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Trace", - "Timestamp": "2026-06-18T09:15:05.500Z", - "Message": "Load Python Script: parse_invoice.py — loaded.", - "ProcessName": "PythonInvoiceParser", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111111-005", - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Error", - "Timestamp": "2026-06-18T09:15:08.610Z", - "Message": "Invoke Python Method: extract_total — UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \"Invoke Python Method: extract_total\". The Python host process exited before returning a result.", - "ProcessName": "PythonInvoiceParser", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/7ebce67886.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/7ebce67886.json deleted file mode 100644 index be95aeb323..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/7ebce67886.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "State": "Pending", - "Timestamp": "2026-06-18T09:15:01.900Z" - }, - { - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "State": "Running", - "Timestamp": "2026-06-18T09:15:02.110Z" - }, - { - "JobKey": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "State": "Faulted", - "Timestamp": "2026-06-18T09:15:08.640Z", - "Info": "UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \"Invoke Python Method: extract_total\"" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/README.md deleted file mode 100644 index 928c1f01e1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Python Scope — Pipe is Broken (Missing pip Module in the Robot Interpreter) - -This scenario reproduces a `Python Scope` failure where `Invoke Python Method` -faults with `Pipe is broken`. The invoked script imports `pandas`, which is -installed in the developer's IDE interpreter but **not** in the Python 3.11 -interpreter the `Python Scope` resolves on the robot host. The job ends with: - -``` -UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: "Invoke Python Method: extract_total" -``` - -## What this scenario uncovers - -**Root Cause:** `UiPath.Python.Activities` runs Python out-of-process and -talks to it over an IPC pipe. `parse_invoice.py` does `import pandas`. The -robot's interpreter (`C:\Program Files\Python311`, the one `Python Scope` -`Path` resolves) does not have `pandas` installed, so the Python host raises -`ModuleNotFoundError` and exits before returning. UiPath only sees the host -die and reports `Pipe is broken` — the underlying Python error is hidden. -The script "runs fine in the developer's IDE" because the IDE uses a -different interpreter that has `pandas`. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/python-activities/playbooks/invoke-python-method-pipe-is-broken.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `PythonInvoiceParser` — `Python Scope` (`Path=C:\Program Files\Python311`, `Target=x64`, `Version=Python_311`) → `Load Python Script: parse_invoice.py` → `Invoke Python Method: extract_total`; `parse_invoice.py` imports `pandas` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `or jobs get` Info shows the `Pipe is broken` `RemoteException` at `Invoke Python Method`, and the logs show the scope opened and the script loaded before the host died | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The decisive evidence is **not** in the job log (the Python traceback is -swallowed): it is the `import pandas` line in `process/parse_invoice.py` -combined with the user's "runs fine in my IDE" framing. The test grades -whether the agent identifies that the out-of-process Python host died on a -missing third-party module in the robot's interpreter, and recommends -installing it into the scope's interpreter / reproducing the real traceback -standalone. - -> **Note on fixtures.** Synthetic. Job key, folder key, and host are -> placeholders representative of a real Robot host layout. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/ccbd9863b4.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/ccbd9863b4.json deleted file mode 100644 index 244ca3c2a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/ccbd9863b4.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "fa111111-2222-3333-4444-555566667777", - "DisplayName": "PythonAutomations", - "FullyQualifiedName": "PythonAutomations", - "Description": "Folder for Python-backed automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/da8db33dfd.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/da8db33dfd.json deleted file mode 100644 index 3d2c2deacc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/da8db33dfd.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-7777-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-06-18T09:15:02.110Z", - "EndTime": "2026-06-18T09:15:08.640Z", - "State": "Faulted", - "ReleaseName": "PythonInvoiceParser", - "Source": "Manual", - "Reference": "InvoiceRunA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "fa111111-2222-3333-4444-555566667777", - "Info": "UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \"Invoke Python Method: extract_total\"", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/manifest.json deleted file mode 100644 index 1313bda589..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/m/r/manifest.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "version": 2, - "_doc": "Canned uip responses for investigating a faulted Python Scope job (folders, jobs, job logs).", - "rules": [ - { "match": "or folders list", "file": "ccbd9863b4.json" }, - { "match": "or jobs list", "file": "da8db33dfd.json" }, - { "match": "or jobs get", "file": "2c90546e13.json" }, - { "match": "or jobs history", "file": "7ebce67886.json" }, - { "match": "or jobs logs", "file": "79723a4720.json" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/uip-fixture.json new file mode 100644 index 0000000000..c3d9c20743 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fa111111-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"PythonAutomations\",\r\n \"FullyQualifiedName\": \"PythonAutomations\",\r\n \"Description\": \"Folder for Python-backed automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-06-18T09:15:02.110Z\",\r\n \"EndTime\": \"2026-06-18T09:15:08.640Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"PythonInvoiceParser\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"InvoiceRunA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"fa111111-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \\\"Invoke Python Method: extract_total\\\"\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-06-18T09:15:02.110Z\",\r\n \"EndTime\": \"2026-06-18T09:15:08.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \\\"Invoke Python Method: extract_total\\\"\\r\\n at UiPath.Python.Service.PythonClient.InvokeMethod(PythonObject instance, String method, IList`1 args)\\r\\n at UiPath.Python.Activities.InvokeMethod.<>c__DisplayClass.b__0()\\r\\n at UiPath.Python.Activities.InvokeMethod.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-06-18T09:15:01.900Z\",\r\n \"ReleaseName\": \"PythonInvoiceParser\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OrganizationUnitId\": 6020001,\r\n \"FolderKey\": \"fa111111-2222-3333-4444-555566667777\",\r\n \"Reference\": \"InvoiceRunA\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"LastModificationTime\": \"2026-06-18T09:15:08.640Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"RemoteException\",\r\n \"ActivityName\": \"InvokeMethod_1\",\r\n \"ActivityDisplayName\": \"Invoke Python Method: extract_total\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Id\": 680078101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-18T09:15:01.900Z\"\r\n },\r\n {\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-18T09:15:02.110Z\"\r\n },\r\n {\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-18T09:15:08.640Z\",\r\n \"Info\": \"UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \\\"Invoke Python Method: extract_total\\\"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa111111-001\",\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-06-18T09:15:02.180Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"PythonInvoiceParser\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-002\",\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-06-18T09:15:02.260Z\",\r\n \"Message\": \"[PythonInvoiceParser] Invoking parse_invoice.extract_total via Python Scope\",\r\n \"ProcessName\": \"PythonInvoiceParser\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-003\",\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-06-18T09:15:04.020Z\",\r\n \"Message\": \"Python Scope — resolved interpreter at 'C:\\\\Program Files\\\\Python311' (Version=Python_311, Target=x64); host process started.\",\r\n \"ProcessName\": \"PythonInvoiceParser\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-004\",\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-06-18T09:15:05.500Z\",\r\n \"Message\": \"Load Python Script: parse_invoice.py — loaded.\",\r\n \"ProcessName\": \"PythonInvoiceParser\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111111-005\",\r\n \"JobKey\": \"aa111111-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-06-18T09:15:08.610Z\",\r\n \"Message\": \"Invoke Python Method: extract_total — UiPath.Python.RemoteException wrapping System.IO.IOException: Pipe is broken: \\\"Invoke Python Method: extract_total\\\". The Python host process exited before returning a result.\",\r\n \"ProcessName\": \"PythonInvoiceParser\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/task.yaml index 8b84aea38f..b8c5ee1464 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-pipe-broken-missing-module/task.yaml @@ -29,21 +29,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/README.md new file mode 100644 index 0000000000..af351e8104 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/README.md @@ -0,0 +1,42 @@ +# Python Scope — One or More Errors Occurred (Target x86 vs 64-bit Install) + +This scenario reproduces a `Python Scope` engine-initialization failure where +the scope's `Target` is `x86` but the Python install on the robot host is +64-bit. The job ends with: + +``` +Python Scope: One or more errors occurred. (Error initializing the Python engine) ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B) +``` + +## What this scenario uncovers + +**Root Cause:** The `Python Scope` `Target` property is set to `x86`, but the +interpreter at `C:\Program Files\Python311` is 64-bit (the user confirms a +64-bit Python 3.11 install). UiPath cannot load a 64-bit interpreter's native +library into a 32-bit host, so engine initialization fails with an aggregate +error whose inner exception is `BadImageFormatException (0x8007000B)` — the +classic 32-bit/64-bit mismatch signature. The fault is at scope open, before +`Run Python Script: crunch.py` runs. + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/python-activities/playbooks/python-scope-architecture-version-mismatch.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `PythonDataCruncher` — `Python Scope` with `Target="x86"`, `Path="C:\Program Files\Python311"`, `Version="Python_311"` → `Run Python Script: crunch.py` (stdlib only) | + +The decisive evidence is `Target="x86"` on the `Python Scope` in +`process/Main.xaml` combined with the user's statement that the installed +Python is 64-bit, plus the `BadImageFormatException (0x8007000B)` inner error. +The script uses only the standard library, ruling out a missing-module cause. +The test grades whether the agent identifies the bitness mismatch and +recommends setting `Target` to `x64`. + +> **Note on fixtures.** Synthetic. Job key, folder key, and host are +> placeholders. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/7194e07d97.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/7194e07d97.json deleted file mode 100644 index 37cf2792a7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/7194e07d97.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "fc333333-2222-3333-4444-555566667777", - "DisplayName": "PythonAutomations", - "FullyQualifiedName": "PythonAutomations", - "Description": "Folder for Python-backed automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020003, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/81a306ef46.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/81a306ef46.json deleted file mode 100644 index 406d652265..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/81a306ef46.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-06-18T13:40:05.200Z", - "EndTime": "2026-06-18T13:40:07.900Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Python Scope: One or more errors occurred. (Error initializing the Python engine) ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)\r\n at UiPath.Python.Service.PythonProcessClient.Start(PythonConfig config)\r\n at UiPath.Python.Activities.PythonScope.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-06-18T13:40:05.000Z", - "ReleaseName": "PythonDataCruncher", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OrganizationUnitId": 6020003, - "FolderKey": "fc333333-2222-3333-4444-555566667777", - "Reference": "CrunchRunA", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "LastModificationTime": "2026-06-18T13:40:07.900Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "AggregateException", - "ActivityName": "PythonScope_1", - "ActivityDisplayName": "Python Scope", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "Id": 680078303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/995237c075.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/995237c075.json deleted file mode 100644 index 8a42f07663..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/995237c075.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-06-18T13:40:05.200Z", - "EndTime": "2026-06-18T13:40:07.900Z", - "State": "Faulted", - "ReleaseName": "PythonDataCruncher", - "Source": "Manual", - "Reference": "CrunchRunA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "fc333333-2222-3333-4444-555566667777", - "Info": "Python Scope: One or more errors occurred. (Error initializing the Python engine)", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/README.md deleted file mode 100644 index b00beda66a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Python Scope — One or More Errors Occurred (Target x86 vs 64-bit Install) - -This scenario reproduces a `Python Scope` engine-initialization failure where -the scope's `Target` is `x86` but the Python install on the robot host is -64-bit. The job ends with: - -``` -Python Scope: One or more errors occurred. (Error initializing the Python engine) ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B) -``` - -## What this scenario uncovers - -**Root Cause:** The `Python Scope` `Target` property is set to `x86`, but the -interpreter at `C:\Program Files\Python311` is 64-bit (the user confirms a -64-bit Python 3.11 install). UiPath cannot load a 64-bit interpreter's native -library into a 32-bit host, so engine initialization fails with an aggregate -error whose inner exception is `BadImageFormatException (0x8007000B)` — the -classic 32-bit/64-bit mismatch signature. The fault is at scope open, before -`Run Python Script: crunch.py` runs. - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/python-activities/playbooks/python-scope-architecture-version-mismatch.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `PythonDataCruncher` — `Python Scope` with `Target="x86"`, `Path="C:\Program Files\Python311"`, `Version="Python_311"` → `Run Python Script: crunch.py` (stdlib only) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `or jobs get` Info shows `One or more errors occurred. (Error initializing the Python engine)` with an inner `BadImageFormatException (0x8007000B)` at `Python Scope` | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The decisive evidence is `Target="x86"` on the `Python Scope` in -`process/Main.xaml` combined with the user's statement that the installed -Python is 64-bit, plus the `BadImageFormatException (0x8007000B)` inner error. -The script uses only the standard library, ruling out a missing-module cause. -The test grades whether the agent identifies the bitness mismatch and -recommends setting `Target` to `x64`. - -> **Note on fixtures.** Synthetic. Job key, folder key, and host are -> placeholders. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/c969796076.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/c969796076.json deleted file mode 100644 index 3aec0415d7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/c969796076.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "JobKey": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "State": "Pending", - "Timestamp": "2026-06-18T13:40:05.000Z" - }, - { - "JobKey": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "State": "Running", - "Timestamp": "2026-06-18T13:40:05.200Z" - }, - { - "JobKey": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "State": "Faulted", - "Timestamp": "2026-06-18T13:40:07.900Z", - "Info": "Python Scope: One or more errors occurred. (Error initializing the Python engine)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/ed55fa9565.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/ed55fa9565.json deleted file mode 100644 index 7bddee751f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/ed55fa9565.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc333333-001", - "JobKey": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-06-18T13:40:05.260Z", - "Message": "Execution started.", - "ProcessName": "PythonDataCruncher", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333333-002", - "JobKey": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-06-18T13:40:05.330Z", - "Message": "[PythonDataCruncher] Opening Python Scope (Target=x86)", - "ProcessName": "PythonDataCruncher", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333333-003", - "JobKey": "cc333333-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Error", - "Timestamp": "2026-06-18T13:40:07.870Z", - "Message": "Python Scope: One or more errors occurred. (Error initializing the Python engine) ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)", - "ProcessName": "PythonDataCruncher", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/manifest.json deleted file mode 100644 index 1a81fb692c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the python-scope-arch-mismatch scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { "match": "or folders list", "file": "7194e07d97.json" }, - { "match": "or jobs list", "file": "995237c075.json" }, - { "match": "or jobs get", "file": "81a306ef46.json" }, - { "match": "or jobs history", "file": "c969796076.json" }, - { "match": "or jobs logs", "file": "ed55fa9565.json" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..30b022b635 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fc333333-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"PythonAutomations\",\r\n \"FullyQualifiedName\": \"PythonAutomations\",\r\n \"Description\": \"Folder for Python-backed automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020003,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-06-18T13:40:05.200Z\",\r\n \"EndTime\": \"2026-06-18T13:40:07.900Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"PythonDataCruncher\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"CrunchRunA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"fc333333-2222-3333-4444-555566667777\",\r\n \"Info\": \"Python Scope: One or more errors occurred. (Error initializing the Python engine)\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-06-18T13:40:05.200Z\",\r\n \"EndTime\": \"2026-06-18T13:40:07.900Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Python Scope: One or more errors occurred. (Error initializing the Python engine) ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)\\r\\n at UiPath.Python.Service.PythonProcessClient.Start(PythonConfig config)\\r\\n at UiPath.Python.Activities.PythonScope.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-06-18T13:40:05.000Z\",\r\n \"ReleaseName\": \"PythonDataCruncher\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OrganizationUnitId\": 6020003,\r\n \"FolderKey\": \"fc333333-2222-3333-4444-555566667777\",\r\n \"Reference\": \"CrunchRunA\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"LastModificationTime\": \"2026-06-18T13:40:07.900Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"AggregateException\",\r\n \"ActivityName\": \"PythonScope_1\",\r\n \"ActivityDisplayName\": \"Python Scope\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Id\": 680078303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"JobKey\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-18T13:40:05.000Z\"\r\n },\r\n {\r\n \"JobKey\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-18T13:40:05.200Z\"\r\n },\r\n {\r\n \"JobKey\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-18T13:40:07.900Z\",\r\n \"Info\": \"Python Scope: One or more errors occurred. (Error initializing the Python engine)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc333333-001\",\r\n \"JobKey\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-06-18T13:40:05.260Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"PythonDataCruncher\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333333-002\",\r\n \"JobKey\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-06-18T13:40:05.330Z\",\r\n \"Message\": \"[PythonDataCruncher] Opening Python Scope (Target=x86)\",\r\n \"ProcessName\": \"PythonDataCruncher\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333333-003\",\r\n \"JobKey\": \"cc333333-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-06-18T13:40:07.870Z\",\r\n \"Message\": \"Python Scope: One or more errors occurred. (Error initializing the Python engine) ---> System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (0x8007000B)\",\r\n \"ProcessName\": \"PythonDataCruncher\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/task.yaml index 4c93a6e9a0..2efe7d4d70 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-arch-mismatch/task.yaml @@ -29,21 +29,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/README.md new file mode 100644 index 0000000000..aaf8a69616 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/README.md @@ -0,0 +1,51 @@ +# Python Scope Hangs / Freezes with No Error - Missing .NET Desktop Runtime + +This scenario reproduces a `Python Scope` that **hangs with no error**. A +Windows-target project on `UiPath.Python.Activities` 1.10.0 runs on a robot host +that is **missing the .NET Desktop Runtime**, so the Python host process never +becomes ready. The job stalls until it is stopped on the max-execution timeout - +with no Python traceback and no engine-init exception. + +## What this scenario uncovers + +**Root Cause:** On `UiPath.Python.Activities` v1.9.0+ with the project target +framework set to **Windows**, the Python host runs on the .NET Desktop Runtime. +The new robot host lacks it, so the host process never starts and the scope +hangs (here, a runtime job that is killed after 15 minutes; the same root cause +freezes Studio at design time). + +This maps to: +`references/activity-packages/python-activities/playbooks/python-scope-hang-dotnet-runtime.md` + +The discriminator vs `python-scope-architecture-version-mismatch.md`: that +playbook keys on an engine-init **error** (bitness / Version / Library path); a +**hang with no error** points at the missing .NET Desktop Runtime. The scope +config here is otherwise valid (Path, `LibraryPath` python311.dll, Target x64), +and the script never runs - so it is not a bitness, Library-path, or script bug. +The user is framed as **off-host**, so the correct agent behavior is to hand +over the Event Viewer / `dotnet --list-runtimes` host check and recommend +installing the runtime (or switching to Windows-Legacy) - not to run host +commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with `targetFramework: Windows`, `UiPath.Python.Activities 1.10.0`, a validly-configured `Python Scope` -> `Load Python Script` -> `Invoke Python Method`, and a clean `scripts/score_model.py` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `python-scope-hang-dotnet-runtime.md` +- Agent identified a missing .NET Desktop Runtime on a Windows-target (v1.9.0+) + project as the cause of the hang (not a bitness / Library-path / script bug) + and recommended installing the .NET Desktop Runtime 6.0 (x64) or switching to + Windows-Legacy, handing over the host check, without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/README.md deleted file mode 100644 index dba787250f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/README.md +++ /dev/null @@ -1,50 +0,0 @@ -# Python Scope Hangs / Freezes with No Error - Missing .NET Desktop Runtime - -This scenario reproduces a `Python Scope` that **hangs with no error**. A -Windows-target project on `UiPath.Python.Activities` 1.10.0 runs on a robot host -that is **missing the .NET Desktop Runtime**, so the Python host process never -becomes ready. The job stalls until it is stopped on the max-execution timeout - -with no Python traceback and no engine-init exception. - -## What this scenario uncovers - -**Root Cause:** On `UiPath.Python.Activities` v1.9.0+ with the project target -framework set to **Windows**, the Python host runs on the .NET Desktop Runtime. -The new robot host lacks it, so the host process never starts and the scope -hangs (here, a runtime job that is killed after 15 minutes; the same root cause -freezes Studio at design time). - -This maps to: -`references/activity-packages/python-activities/playbooks/python-scope-hang-dotnet-runtime.md` - -The discriminator vs `python-scope-architecture-version-mismatch.md`: that -playbook keys on an engine-init **error** (bitness / Version / Library path); a -**hang with no error** points at the missing .NET Desktop Runtime. The scope -config here is otherwise valid (Path, `LibraryPath` python311.dll, Target x64), -and the script never runs - so it is not a bitness, Library-path, or script bug. -The user is framed as **off-host**, so the correct agent behavior is to hand -over the Event Viewer / `dotnet --list-runtimes` host check and recommend -installing the runtime (or switching to Windows-Legacy) - not to run host -commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with `targetFramework: Windows`, `UiPath.Python.Activities 1.10.0`, a validly-configured `Python Scope` -> `Load Python Script` -> `Invoke Python Method`, and a clean `scripts/score_model.py` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature (a ~15-minute hang faulted on timeout, no Python error) | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `python-scope-hang-dotnet-runtime.md` -- Agent identified a missing .NET Desktop Runtime on a Windows-target (v1.9.0+) - project as the cause of the hang (not a bitness / Library-path / script bug) - and recommended installing the .NET Desktop Runtime 6.0 (x64) or switching to - Windows-Legacy, handing over the host check, without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/manifest.json deleted file mode 100644 index 21e1c9d68c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the documented error signature.", - "rules": [ - { - "match": "or folders list --output json", - "file": "or-folders-list-output-json.json" - }, - { - "match": "or jobs list --folder-key \"cc01a2b3-d4e5-4f60-8c01-000000000001\" --state Faulted --output json", - "file": "or-jobs-list-folder-key-cc01a2b3-state-faulted-output-json.json" - }, - { - "match": "or jobs list --folder-key cc01a2b3-d4e5-4f60-8c01-000000000001 --state Faulted --output json", - "file": "or-jobs-list-folder-key-cc01a2b3-state-faulted-output-json.json" - }, - { - "match": "or jobs get \"dd01b2c3-e5f6-4071-9d01-000000000001\" --output json", - "file": "or-jobs-get-dd01b2c3-output-json.json" - }, - { - "match": "or jobs get dd01b2c3-e5f6-4071-9d01-000000000001 --output json", - "file": "or-jobs-get-dd01b2c3-output-json.json" - }, - { - "match": "or jobs logs \"dd01b2c3-e5f6-4071-9d01-000000000001\" --level Error --output json", - "file": "or-jobs-logs-dd01b2c3-level-error-output-json.json" - }, - { - "match": "or jobs logs dd01b2c3-e5f6-4071-9d01-000000000001 --level Error --output json", - "file": "or-jobs-logs-dd01b2c3-level-error-output-json.json" - }, - { - "match": "or jobs logs \"dd01b2c3-e5f6-4071-9d01-000000000001\" --output json", - "file": "or-jobs-logs-dd01b2c3-output-json.json" - }, - { - "match": "or jobs logs dd01b2c3-e5f6-4071-9d01-000000000001 --output json", - "file": "or-jobs-logs-dd01b2c3-output-json.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-folders-list-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-folders-list-output-json.json deleted file mode 100644 index 7ebd41aaf9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-folders-list-output-json.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "cc01a2b3-d4e5-4f60-8c01-000000000001", - "Name": "ML Pipelines", - "Path": "ML Pipelines", - "Description": "Production folder for model-scoring processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-get-dd01b2c3-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-get-dd01b2c3-output-json.json deleted file mode 100644 index c0b2e55e92..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-get-dd01b2c3-output-json.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dd01b2c3-e5f6-4071-9d01-000000000001", - "StartTime": "2026-06-12T09:05:11.210Z", - "EndTime": "2026-06-12T09:20:12.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ee01c3d4-f6a7-4182-9e01-000000000001", - "Info": "Python Scope: The activity did not complete. The Python host process did not respond and the job was stopped after exceeding its maximum execution time (15 minutes). No error was reported by the Python side.\r\n\r\nin Main.xaml\r\n at PythonScope \"Python Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"", - "CreationTime": "2026-06-12T09:05:10.800Z", - "ReleaseName": "PyScoreModel", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 7012501, - "OrganizationUnitId": 7791618, - "FolderKey": "cc01a2b3-d4e5-4f60-8c01-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T09:20:12.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "dd01b2c3-e5f6-4071-9d01-000000000001", - "Id": 712450501 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-list-folder-key-cc01a2b3-state-faulted-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-list-folder-key-cc01a2b3-state-faulted-output-json.json deleted file mode 100644 index 9c99b4a5b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-list-folder-key-cc01a2b3-state-faulted-output-json.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "dd01b2c3-e5f6-4071-9d01-000000000001", - "State": "Faulted", - "ProcessName": "PyScoreModel", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT", - "MachineKey": "", - "StartTime": "2026-06-12T09:05:11.210Z", - "EndTime": "2026-06-12T09:20:12.880Z", - "CreationTime": "2026-06-12T09:05:10.800Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-logs-dd01b2c3-level-error-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-logs-dd01b2c3-level-error-output-json.json deleted file mode 100644 index b35f2e399a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-logs-dd01b2c3-level-error-output-json.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-12T09:20:12.640Z", - "Message": "[Python Scope] The Python host process did not respond. The job was stopped after exceeding its maximum execution time (15 minutes). No error was reported by the Python side." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-logs-dd01b2c3-output-json.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-logs-dd01b2c3-output-json.json deleted file mode 100644 index 473be8eb73..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/m/r/or-jobs-logs-dd01b2c3-output-json.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-12T09:05:11.240Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-12T09:05:12.180Z", - "Message": "[Python Scope] Starting the Python host process (Target: x64, Path: C:\\Python311, LibraryPath: C:\\Python311\\python311.dll, Version: Auto)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-12T09:05:12.190Z", - "Message": "[Python Scope] Waiting for the Python host process to become ready..." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-12T09:20:12.640Z", - "Message": "[Python Scope] The Python host process did not respond. The job was stopped after exceeding its maximum execution time (15 minutes). No error was reported by the Python side." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-12T09:20:12.820Z", - "Message": "Execution ended in: 00:15:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/uip-fixture.json new file mode 100644 index 0000000000..40d372afd9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "cc01a2b3-d4e5-4f60-8c01-000000000001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dd01b2c3-e5f6-4071-9d01-000000000001\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PyScoreModel\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T09:05:11.210Z\",\r\n \"EndTime\": \"2026-06-12T09:20:12.880Z\",\r\n \"CreationTime\": \"2026-06-12T09:05:10.800Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dd01b2c3-e5f6-4071-9d01-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T09:20:12.640Z\",\r\n \"Message\": \"[Python Scope] The Python host process did not respond. The job was stopped after exceeding its maximum execution time (15 minutes). No error was reported by the Python side.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "dd01b2c3-e5f6-4071-9d01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dd01b2c3-e5f6-4071-9d01-000000000001\",\r\n \"StartTime\": \"2026-06-12T09:05:11.210Z\",\r\n \"EndTime\": \"2026-06-12T09:20:12.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ee01c3d4-f6a7-4182-9e01-000000000001\",\r\n \"Info\": \"Python Scope: The activity did not complete. The Python host process did not respond and the job was stopped after exceeding its maximum execution time (15 minutes). No error was reported by the Python side.\\r\\n\\r\\nin Main.xaml\\r\\n at PythonScope \\\"Python Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\",\r\n \"CreationTime\": \"2026-06-12T09:05:10.800Z\",\r\n \"ReleaseName\": \"PyScoreModel\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 7012501,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"cc01a2b3-d4e5-4f60-8c01-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T09:20:12.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"dd01b2c3-e5f6-4071-9d01-000000000001\",\r\n \"Id\": 712450501\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dd01b2c3-e5f6-4071-9d01-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-12T09:05:11.240Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-12T09:05:12.180Z\",\r\n \"Message\": \"[Python Scope] Starting the Python host process (Target: x64, Path: C:\\\\Python311, LibraryPath: C:\\\\Python311\\\\python311.dll, Version: Auto)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-12T09:05:12.190Z\",\r\n \"Message\": \"[Python Scope] Waiting for the Python host process to become ready...\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T09:20:12.640Z\",\r\n \"Message\": \"[Python Scope] The Python host process did not respond. The job was stopped after exceeding its maximum execution time (15 minutes). No error was reported by the Python side.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-12T09:20:12.820Z\",\r\n \"Message\": \"Execution ended in: 00:15:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc01a2b3-d4e5-4f60-8c01-000000000001\",\r\n \"Name\": \"ML Pipelines\",\r\n \"Path\": \"ML Pipelines\",\r\n \"Description\": \"Production folder for model-scoring processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/task.yaml index a757fa3609..baab4860e9 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-dotnet-runtime-hang/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/README.md new file mode 100644 index 0000000000..370b8b0306 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/README.md @@ -0,0 +1,40 @@ +# Python Scope — The Specified Python Path Is Not Valid (Path Points at python.exe) + +This scenario reproduces a `Python Scope` failure where the scope cannot +resolve the interpreter because its `Path` property points at `python.exe` +(the executable) instead of the install **folder**. The job ends with: + +``` +Python Scope: The specified Python path is not valid: C:\Program Files\Python311\python.exe +``` + +## What this scenario uncovers + +**Root Cause:** The `Python Scope` `Path` property must be the **folder** that +contains `python.exe` (e.g. `C:\Program Files\Python311`). It is set to +`C:\Program Files\Python311\python.exe` — the executable itself. The scope +appends the interpreter name internally, so it cannot resolve a real +interpreter from a path that already ends in `python.exe`, and fails at scope +open before any Python runs. The named file does exist on disk, which is why +the user thinks "the path is right." + +This maps to: +`skills/uipath-troubleshoot/references/activity-packages/python-activities/playbooks/python-path-not-valid.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `PythonHelloWorld` — `Python Scope` with `Path="C:\Program Files\Python311\python.exe"` (trailing `\python.exe`) → `Run Python Script: hello.py` | + +The decisive evidence is the trailing `\python.exe` on the `Python Scope` +`Path` in `process/Main.xaml`, matching the path echoed in the error. The +test grades whether the agent identifies the file-vs-folder mistake and +recommends stripping `\python.exe` to the containing folder. + +> **Note on fixtures.** Synthetic. Job key, folder key, and host are +> placeholders. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/12ead3e479.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/12ead3e479.json deleted file mode 100644 index ebf8853969..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/12ead3e479.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "JobKey": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "State": "Pending", - "Timestamp": "2026-06-18T11:02:10.100Z" - }, - { - "JobKey": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "State": "Running", - "Timestamp": "2026-06-18T11:02:10.300Z" - }, - { - "JobKey": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "State": "Faulted", - "Timestamp": "2026-06-18T11:02:12.180Z", - "Info": "Python Scope: The specified Python path is not valid: C:\\Program Files\\Python311\\python.exe" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/348d600ec1.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/348d600ec1.json deleted file mode 100644 index 8976abfe8b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/348d600ec1.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-06-18T11:02:10.300Z", - "EndTime": "2026-06-18T11:02:12.180Z", - "State": "Faulted", - "ReleaseName": "PythonHelloWorld", - "Source": "Manual", - "Reference": "HelloRunA", - "HostMachineName": "MOCK-HOST", - "FolderKey": "fb222222-2222-3333-4444-555566667777", - "Info": "Python Scope: The specified Python path is not valid: C:\\Program Files\\Python311\\python.exe", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/6ed334f24b.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/6ed334f24b.json deleted file mode 100644 index c1818ff161..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/6ed334f24b.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "fb222222-2222-3333-4444-555566667777", - "DisplayName": "PythonAutomations", - "FullyQualifiedName": "PythonAutomations", - "Description": "Folder for Python-backed automations.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6020002, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/9a87aba928.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/9a87aba928.json deleted file mode 100644 index a770c6796d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/9a87aba928.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-bb222222-001", - "JobKey": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-06-18T11:02:10.360Z", - "Message": "Execution started.", - "ProcessName": "PythonHelloWorld", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222222-002", - "JobKey": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-06-18T11:02:10.440Z", - "Message": "[PythonHelloWorld] Opening Python Scope", - "ProcessName": "PythonHelloWorld", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222222-003", - "JobKey": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "Level": "Error", - "Timestamp": "2026-06-18T11:02:12.150Z", - "Message": "Python Scope: The specified Python path is not valid: C:\\Program Files\\Python311\\python.exe", - "ProcessName": "PythonHelloWorld", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/README.md deleted file mode 100644 index 33fdabbe59..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Python Scope — The Specified Python Path Is Not Valid (Path Points at python.exe) - -This scenario reproduces a `Python Scope` failure where the scope cannot -resolve the interpreter because its `Path` property points at `python.exe` -(the executable) instead of the install **folder**. The job ends with: - -``` -Python Scope: The specified Python path is not valid: C:\Program Files\Python311\python.exe -``` - -## What this scenario uncovers - -**Root Cause:** The `Python Scope` `Path` property must be the **folder** that -contains `python.exe` (e.g. `C:\Program Files\Python311`). It is set to -`C:\Program Files\Python311\python.exe` — the executable itself. The scope -appends the interpreter name internally, so it cannot resolve a real -interpreter from a path that already ends in `python.exe`, and fails at scope -open before any Python runs. The named file does exist on disk, which is why -the user thinks "the path is right." - -This maps to: -`skills/uipath-troubleshoot/references/activity-packages/python-activities/playbooks/python-path-not-valid.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | `PythonHelloWorld` — `Python Scope` with `Path="C:\Program Files\Python311\python.exe"` (trailing `\python.exe`) → `Run Python Script: hello.py` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `or jobs get` Info shows `The specified Python path is not valid: C:\Program Files\Python311\python.exe` at `Python Scope`, faulting at scope open | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The decisive evidence is the trailing `\python.exe` on the `Python Scope` -`Path` in `process/Main.xaml`, matching the path echoed in the error. The -test grades whether the agent identifies the file-vs-folder mistake and -recommends stripping `\python.exe` to the containing folder. - -> **Note on fixtures.** Synthetic. Job key, folder key, and host are -> placeholders. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/b402d7b9ea.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/b402d7b9ea.json deleted file mode 100644 index ec5de347e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/b402d7b9ea.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-06-18T11:02:10.300Z", - "EndTime": "2026-06-18T11:02:12.180Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Python Scope: The specified Python path is not valid: C:\\Program Files\\Python311\\python.exe\r\n at UiPath.Python.Service.PythonClient.Initialize(PythonConfig config)\r\n at UiPath.Python.Activities.PythonScope.OnExecuteAsync(NativeActivityContext context)\r\n", - "CreationTime": "2026-06-18T11:02:10.100Z", - "ReleaseName": "PythonHelloWorld", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OrganizationUnitId": 6020002, - "FolderKey": "fb222222-2222-3333-4444-555566667777", - "Reference": "HelloRunA", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "LastModificationTime": "2026-06-18T11:02:12.180Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidOperationException", - "ActivityName": "PythonScope_1", - "ActivityDisplayName": "Python Scope", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "bb222222-7777-bbbb-cccc-ddddeeeeffff", - "Id": 680078202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/manifest.json deleted file mode 100644 index 8aade09585..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/m/r/manifest.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the python-scope-path-is-file scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { "match": "or folders list", "file": "6ed334f24b.json" }, - { "match": "or jobs list", "file": "348d600ec1.json" }, - { "match": "or jobs get", "file": "b402d7b9ea.json" }, - { "match": "or jobs history", "file": "12ead3e479.json" }, - { "match": "or jobs logs", "file": "9a87aba928.json" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/uip-fixture.json new file mode 100644 index 0000000000..bab60118be --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"fb222222-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"PythonAutomations\",\r\n \"FullyQualifiedName\": \"PythonAutomations\",\r\n \"Description\": \"Folder for Python-backed automations.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6020002,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-06-18T11:02:10.300Z\",\r\n \"EndTime\": \"2026-06-18T11:02:12.180Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"PythonHelloWorld\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"HelloRunA\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"fb222222-2222-3333-4444-555566667777\",\r\n \"Info\": \"Python Scope: The specified Python path is not valid: C:\\\\Program Files\\\\Python311\\\\python.exe\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-06-18T11:02:10.300Z\",\r\n \"EndTime\": \"2026-06-18T11:02:12.180Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Python Scope: The specified Python path is not valid: C:\\\\Program Files\\\\Python311\\\\python.exe\\r\\n at UiPath.Python.Service.PythonClient.Initialize(PythonConfig config)\\r\\n at UiPath.Python.Activities.PythonScope.OnExecuteAsync(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-06-18T11:02:10.100Z\",\r\n \"ReleaseName\": \"PythonHelloWorld\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OrganizationUnitId\": 6020002,\r\n \"FolderKey\": \"fb222222-2222-3333-4444-555566667777\",\r\n \"Reference\": \"HelloRunA\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"LastModificationTime\": \"2026-06-18T11:02:12.180Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidOperationException\",\r\n \"ActivityName\": \"PythonScope_1\",\r\n \"ActivityDisplayName\": \"Python Scope\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Id\": 680078202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"JobKey\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-18T11:02:10.100Z\"\r\n },\r\n {\r\n \"JobKey\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-18T11:02:10.300Z\"\r\n },\r\n {\r\n \"JobKey\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-18T11:02:12.180Z\",\r\n \"Info\": \"Python Scope: The specified Python path is not valid: C:\\\\Program Files\\\\Python311\\\\python.exe\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-bb222222-001\",\r\n \"JobKey\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-06-18T11:02:10.360Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"PythonHelloWorld\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222222-002\",\r\n \"JobKey\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-06-18T11:02:10.440Z\",\r\n \"Message\": \"[PythonHelloWorld] Opening Python Scope\",\r\n \"ProcessName\": \"PythonHelloWorld\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222222-003\",\r\n \"JobKey\": \"bb222222-7777-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-06-18T11:02:12.150Z\",\r\n \"Message\": \"Python Scope: The specified Python path is not valid: C:\\\\Program Files\\\\Python311\\\\python.exe\",\r\n \"ProcessName\": \"PythonHelloWorld\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/task.yaml index 9763db1b52..abfa2581a8 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/py-scope-path-is-file/task.yaml @@ -27,21 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/README.md new file mode 100644 index 0000000000..2ca7655398 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/README.md @@ -0,0 +1,30 @@ +# SAP Application Scope — invalid advanced parameter (SapActivityException, before connect) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the invalid-advanced-parameter case of `UiPath.SAP.BAPI.Activities.SapApplicationScope` — a distinct exception class from the no-host case. + +## What this exercises + +A `SAP Application Scope` has an unsupported RFC key in its Advanced Parameters. The UiPath connection layer parses the advanced parameters and rejects the connection before any network call. The job ends Faulted with `UiPath.SAP.BAPI.Utilities.SapActivityException: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.` (from `SapConnectionService.OverideWithAdvancedParameters`). The agent must read the **invalid-advanced-parameter signature** as the cause — not blame the BAPI (none ran), a missing host (`RfcConfigurationException`), a SAP outage (`RfcCommunicationException`), logon (`RfcLogonException`), or a workflow bug. The fix is to remove/correct the advanced parameter. + +Signature captured **verbatim from a live `uip rpa` run** of a `SapApplicationScope` with `AdvancedParameters="BOGUSKEY=abc"` (no SAP system — the advanced-param check runs before any network call). Workflow/process names neutralized. Maps to the [sap-connection-failed](../../../../../skills/uipath-troubleshoot/references/activity-packages/sap-bapi-activities/playbooks/sap-connection-failed.md) playbook (the `SapActivityException` advanced-parameter branch). + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, SapActivityException: Advanced Parameters has invalid parameter …) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the config error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/0a92c3205f.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/0a92c3205f.json deleted file mode 100644 index dd19aaf932..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/0a92c3205f.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "StartTime": "2026-06-20T05:40:18.220Z", - "EndTime": "2026-06-20T05:40:19.910Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c3d4e5f6-a7b8-4c9d-8e0f-1a2b3c4d5e6f", - "Info": "Source: SAP Application Scope\r\nMessage: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\r\nException Type: UiPath.SAP.BAPI.Utilities.SapActivityException\r\nin Main.xaml\r\n at SapApplicationScope \"SAP Application Scope\"\r\n at Main \"Main\"\r\n\r\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.OverideWithAdvancedParameters(SapConnection sapConnection)\r\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\r\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Connect(NativeActivityContext context)\r\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T05:40:17.050Z", - "ReleaseName": "SapMasterDataLoad", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1107880, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T05:40:19.910Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "2e7c9a104b6d4f8291c35a8e0d1f2b30", - "Id": 50605140 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/6a8b362eb6.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/6a8b362eb6.json deleted file mode 100644 index 5569ca1bf9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/6a8b362eb6.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T05:40:19.700Z", - "Message": "SAP Application Scope: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\r\nException Type: UiPath.SAP.BAPI.Utilities.SapActivityException\r\nin Main.xaml\r\n at SapApplicationScope \"SAP Application Scope\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.SAP.BAPI.Utilities.SapActivityException: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\r\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.OverideWithAdvancedParameters(SapConnection sapConnection)\r\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/README.md deleted file mode 100644 index 6aafafbbd9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# SAP Application Scope — invalid advanced parameter (SapActivityException, before connect) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the invalid-advanced-parameter case of `UiPath.SAP.BAPI.Activities.SapApplicationScope` — a distinct exception class from the no-host case. - -## What this exercises - -A `SAP Application Scope` has an unsupported RFC key in its Advanced Parameters. The UiPath connection layer parses the advanced parameters and rejects the connection before any network call. The job ends Faulted with `UiPath.SAP.BAPI.Utilities.SapActivityException: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.` (from `SapConnectionService.OverideWithAdvancedParameters`). The agent must read the **invalid-advanced-parameter signature** as the cause — not blame the BAPI (none ran), a missing host (`RfcConfigurationException`), a SAP outage (`RfcCommunicationException`), logon (`RfcLogonException`), or a workflow bug. The fix is to remove/correct the advanced parameter. - -Signature captured **verbatim from a live `uip rpa` run** of a `SapApplicationScope` with `AdvancedParameters="BOGUSKEY=abc"` (no SAP system — the advanced-param check runs before any network call). Workflow/process names neutralized. Maps to the [sap-connection-failed](../../../../../skills/uipath-troubleshoot/references/activity-packages/sap-bapi-activities/playbooks/sap-connection-failed.md) playbook (the `SapActivityException` advanced-parameter branch). - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, SapActivityException: Advanced Parameters has invalid parameter …) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the config error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/b1e0d5aec7.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/b1e0d5aec7.json deleted file mode 100644 index 04b5c82508..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/b1e0d5aec7.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "State": "Faulted", - "StartTime": "2026-06-20T05:40:18.220Z", - "EndTime": "2026-06-20T05:40:19.910Z", - "ReleaseName": "SapMasterDataLoad", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50605140 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/manifest.json deleted file mode 100644 index bdbf870fd4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.SAP.BAPI.Activities SAP Application Scope faults with UiPath.SAP.BAPI.Utilities.SapActivityException 'Advanced Parameters has invalid parameter . Connection cannot be established.' (from SapConnectionService.OverideWithAdvancedParameters) because an unsupported RFC advanced-parameter key was configured. Signature captured verbatim from a live uip rpa run (no SAP system - fails at advanced-param parsing before any network call); workflow/process names neutralized. The agent must read the invalid advanced-parameter as the cause (a UiPath-layer config check before connect), not blame the BAPI, host reachability, or logon. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "b1e0d5aec7.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "b1e0d5aec7.json" - }, - { - "match": "or jobs list", - "file": "b1e0d5aec7.json" - }, - { - "match": "or jobs get 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "file": "0a92c3205f.json" - }, - { - "match": "or jobs logs 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30 --level Error", - "file": "6a8b362eb6.json" - }, - { - "match": "or jobs logs 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "file": "6a8b362eb6.json" - }, - { - "match": "or jobs traces 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in Shared to find the failing job." - }, - { - "pattern": "or jobs get 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", - "min": 1, - "description": "Agent fetches the faulted job to read the SapActivityException invalid-advanced-parameter signature." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/uip-fixture.json new file mode 100644 index 0000000000..18bfed6522 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T05:40:18.220Z\",\r\n \"EndTime\": \"2026-06-20T05:40:19.910Z\",\r\n \"ReleaseName\": \"SapMasterDataLoad\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50605140\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T05:40:19.700Z\",\r\n \"Message\": \"SAP Application Scope: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\\r\\nException Type: UiPath.SAP.BAPI.Utilities.SapActivityException\\r\\nin Main.xaml\\r\\n at SapApplicationScope \\\"SAP Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.SAP.BAPI.Utilities.SapActivityException: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.OverideWithAdvancedParameters(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T05:40:18.220Z\",\r\n \"EndTime\": \"2026-06-20T05:40:19.910Z\",\r\n \"ReleaseName\": \"SapMasterDataLoad\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50605140\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30\",\r\n \"StartTime\": \"2026-06-20T05:40:18.220Z\",\r\n \"EndTime\": \"2026-06-20T05:40:19.910Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c3d4e5f6-a7b8-4c9d-8e0f-1a2b3c4d5e6f\",\r\n \"Info\": \"Source: SAP Application Scope\\r\\nMessage: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\\r\\nException Type: UiPath.SAP.BAPI.Utilities.SapActivityException\\r\\nin Main.xaml\\r\\n at SapApplicationScope \\\"SAP Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.OverideWithAdvancedParameters(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Connect(NativeActivityContext context)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T05:40:17.050Z\",\r\n \"ReleaseName\": \"SapMasterDataLoad\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1107880,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T05:40:19.910Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"2e7c9a104b6d4f8291c35a8e0d1f2b30\",\r\n \"Id\": 50605140\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T05:40:19.700Z\",\r\n \"Message\": \"SAP Application Scope: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\\r\\nException Type: UiPath.SAP.BAPI.Utilities.SapActivityException\\r\\nin Main.xaml\\r\\n at SapApplicationScope \\\"SAP Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.SAP.BAPI.Utilities.SapActivityException: Advanced Parameters has invalid parameter BOGUSKEY. Connection cannot be established.\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.OverideWithAdvancedParameters(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T05:40:18.220Z\",\r\n \"EndTime\": \"2026-06-20T05:40:19.910Z\",\r\n \"ReleaseName\": \"SapMasterDataLoad\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50605140\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in Shared to find the failing job." + }, + { + "pattern": "or jobs get 2e7c9a10-4b6d-4f82-91c3-5a8e0d1f2b30", + "min": 1, + "description": "Agent fetches the faulted job to read the SapActivityException invalid-advanced-parameter signature." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/task.yaml index 38f50c70a4..bc515348ce 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-invalid-advanced-param/task.yaml @@ -22,20 +22,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/README.md new file mode 100644 index 0000000000..c11639cf8e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/README.md @@ -0,0 +1,30 @@ +# SAP Application Scope — no connection configured (RfcConfigurationException, not a BAPI or outage) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the connection-configuration failure of `UiPath.SAP.BAPI.Activities.SapApplicationScope`. + +## What this exercises + +A `SAP Application Scope` faults while opening the RFC connection because no connection parameters (application/message server) were configured. The job ends Faulted with `SAP.Middleware.Connector.RfcConfigurationException: Not any kind of host is specified` (from `RfcDestination.CheckConfiguration()` via `SapConnectionService.Connect`), before any BAPI or network call. The agent must read the **connection-config signature** as the cause (missing SAP connection parameters) — not blame the BAPI (none ran), a live SAP outage (`RfcCommunicationException`), a logon problem (`RfcLogonException`), or a workflow-logic bug. The fix is to configure the Application/Message Server + client/user/password. + +Signature captured **verbatim from a live `uip rpa` run** of a `SapApplicationScope` with empty connection parameters (no SAP system needed — the SAP .NET Connector rejects the config before any network call). Workflow/process names neutralized. Maps to the [sap-connection-failed](../../../../../skills/uipath-troubleshoot/references/activity-packages/sap-bapi-activities/playbooks/sap-connection-failed.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, RfcConfigurationException: Not any kind of host is specified) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the config error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/9733e14175.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/9733e14175.json deleted file mode 100644 index 95f831c107..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/9733e14175.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "State": "Faulted", - "StartTime": "2026-06-20T06:22:41.510Z", - "EndTime": "2026-06-20T06:22:43.980Z", - "ReleaseName": "SapOrderSync", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50607233 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/README.md deleted file mode 100644 index 799185480b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# SAP Application Scope — no connection configured (RfcConfigurationException, not a BAPI or outage) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the connection-configuration failure of `UiPath.SAP.BAPI.Activities.SapApplicationScope`. - -## What this exercises - -A `SAP Application Scope` faults while opening the RFC connection because no connection parameters (application/message server) were configured. The job ends Faulted with `SAP.Middleware.Connector.RfcConfigurationException: Not any kind of host is specified` (from `RfcDestination.CheckConfiguration()` via `SapConnectionService.Connect`), before any BAPI or network call. The agent must read the **connection-config signature** as the cause (missing SAP connection parameters) — not blame the BAPI (none ran), a live SAP outage (`RfcCommunicationException`), a logon problem (`RfcLogonException`), or a workflow-logic bug. The fix is to configure the Application/Message Server + client/user/password. - -Signature captured **verbatim from a live `uip rpa` run** of a `SapApplicationScope` with empty connection parameters (no SAP system needed — the SAP .NET Connector rejects the config before any network call). Workflow/process names neutralized. Maps to the [sap-connection-failed](../../../../../skills/uipath-troubleshoot/references/activity-packages/sap-bapi-activities/playbooks/sap-connection-failed.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, RfcConfigurationException: Not any kind of host is specified) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the config error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/bf638d2c85.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/bf638d2c85.json deleted file mode 100644 index ca6360866f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/bf638d2c85.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T06:22:43.700Z", - "Message": "SAP Application Scope: Not any kind of host is specified\r\nException Type: SAP.Middleware.Connector.RfcConfigurationException\r\nin Main.xaml\r\n at SapApplicationScope \"SAP Application Scope\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSAP.Middleware.Connector.RfcConfigurationException: Not any kind of host is specified\r\n at SAP.Middleware.Connector.RfcDestination.CheckConfiguration()\r\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\r\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/e3a37e60eb.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/e3a37e60eb.json deleted file mode 100644 index c41ff67097..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/e3a37e60eb.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "StartTime": "2026-06-20T06:22:41.510Z", - "EndTime": "2026-06-20T06:22:43.980Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b0c1d2e3-f4a5-4b6c-8d7e-9f0a1b2c3d4e", - "Info": "Source: SAP Application Scope\r\nMessage: Not any kind of host is specified\r\nException Type: SAP.Middleware.Connector.RfcConfigurationException\r\nin Main.xaml\r\n at SapApplicationScope \"SAP Application Scope\"\r\n at Main \"Main\"\r\n\r\n at SAP.Middleware.Connector.RfcDestination.CheckConfiguration()\r\n at SAP.Middleware.Connector.RfcDestinationManager.GetDestination(String destinationName)\r\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnector.SetRepository(String destinationNameString)\r\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\r\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Connect(NativeActivityContext context)\r\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T06:22:40.300Z", - "ReleaseName": "SapOrderSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108102, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T06:22:43.980Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "4f8a1c607e2d4b93a5c16d3e9f0a2b40", - "Id": 50607233 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/manifest.json deleted file mode 100644 index 5ab823fb04..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a UiPath.SAP.BAPI.Activities SAP Application Scope faults opening the RFC connection with SAP.Middleware.Connector.RfcConfigurationException 'Not any kind of host is specified' (from RfcDestination.CheckConfiguration via SapConnectionService.Connect) because no connection parameters were configured. Signature captured verbatim from a live uip rpa run (no SAP system needed - fails at config before any network call); workflow/process names neutralized. The agent must read the connection-config signature as the cause (missing connection params/server), before any BAPI, not blame the BAPI or a live SAP outage. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "9733e14175.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "9733e14175.json" - }, - { - "match": "or jobs list", - "file": "9733e14175.json" - }, - { - "match": "or jobs get 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "file": "e3a37e60eb.json" - }, - { - "match": "or jobs logs 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40 --level Error", - "file": "bf638d2c85.json" - }, - { - "match": "or jobs logs 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "file": "bf638d2c85.json" - }, - { - "match": "or jobs traces 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in Shared to find the failing job." - }, - { - "pattern": "or jobs get 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", - "min": 1, - "description": "Agent fetches the faulted job to read the RfcConfigurationException 'Not any kind of host is specified' connection-config signature." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/uip-fixture.json new file mode 100644 index 0000000000..ef70095dcc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T06:22:41.510Z\",\r\n \"EndTime\": \"2026-06-20T06:22:43.980Z\",\r\n \"ReleaseName\": \"SapOrderSync\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50607233\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T06:22:43.700Z\",\r\n \"Message\": \"SAP Application Scope: Not any kind of host is specified\\r\\nException Type: SAP.Middleware.Connector.RfcConfigurationException\\r\\nin Main.xaml\\r\\n at SapApplicationScope \\\"SAP Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSAP.Middleware.Connector.RfcConfigurationException: Not any kind of host is specified\\r\\n at SAP.Middleware.Connector.RfcDestination.CheckConfiguration()\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T06:22:41.510Z\",\r\n \"EndTime\": \"2026-06-20T06:22:43.980Z\",\r\n \"ReleaseName\": \"SapOrderSync\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50607233\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40\",\r\n \"StartTime\": \"2026-06-20T06:22:41.510Z\",\r\n \"EndTime\": \"2026-06-20T06:22:43.980Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b0c1d2e3-f4a5-4b6c-8d7e-9f0a1b2c3d4e\",\r\n \"Info\": \"Source: SAP Application Scope\\r\\nMessage: Not any kind of host is specified\\r\\nException Type: SAP.Middleware.Connector.RfcConfigurationException\\r\\nin Main.xaml\\r\\n at SapApplicationScope \\\"SAP Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n at SAP.Middleware.Connector.RfcDestination.CheckConfiguration()\\r\\n at SAP.Middleware.Connector.RfcDestinationManager.GetDestination(String destinationName)\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnector.SetRepository(String destinationNameString)\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Connect(NativeActivityContext context)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T06:22:40.300Z\",\r\n \"ReleaseName\": \"SapOrderSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108102,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T06:22:43.980Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"4f8a1c607e2d4b93a5c16d3e9f0a2b40\",\r\n \"Id\": 50607233\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T06:22:43.700Z\",\r\n \"Message\": \"SAP Application Scope: Not any kind of host is specified\\r\\nException Type: SAP.Middleware.Connector.RfcConfigurationException\\r\\nin Main.xaml\\r\\n at SapApplicationScope \\\"SAP Application Scope\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSAP.Middleware.Connector.RfcConfigurationException: Not any kind of host is specified\\r\\n at SAP.Middleware.Connector.RfcDestination.CheckConfiguration()\\r\\n at UiPath.SAP.BAPI.ConnectionService.SapConnectionService.SapConnectionService.Connect(SapConnection sapConnection)\\r\\n at UiPath.SAP.BAPI.Activities.SapApplicationScope.Execute(NativeActivityContext context)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T06:22:41.510Z\",\r\n \"EndTime\": \"2026-06-20T06:22:43.980Z\",\r\n \"ReleaseName\": \"SapOrderSync\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50607233\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in Shared to find the failing job." + }, + { + "pattern": "or jobs get 4f8a1c60-7e2d-4b93-a5c1-6d3e9f0a2b40", + "min": 1, + "description": "Agent fetches the faulted job to read the RfcConfigurationException 'Not any kind of host is specified' connection-config signature." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/task.yaml index d6279eec9d..5431511b38 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sap-connection-no-host/task.yaml @@ -21,20 +21,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/README.md new file mode 100644 index 0000000000..edf6a80f02 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/README.md @@ -0,0 +1,30 @@ +# Slack Send Message — channel_not_found (read the ProviderMessage, not the HTTP 400) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the `UiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException` case of `UiPath.Slack.IntegrationService.Activities.SendMessage`. + +## What this exercises + +A Slack **Send Message** ("Send Message to Channel") faults *after* its Integration Service connection resolves, because the Slack Web API rejects the call. The job ends Faulted with `BusinessActivityExecutionException` carrying `Error Code: [400]` and the raw Slack `ProviderMessage: {"ok":false,"error":"channel_not_found",...}`. The agent must read the **Slack `error` code** (`channel_not_found`) as the cause — not stop at the generic HTTP 400, and not blame connection resolution (that would be a `System.AggregateException` / `ConnectionException`, a different playbook). The fix is to correct the Channel argument. + +Signature captured verbatim from a real Orchestrator job produced by running a Slack `SendMessage` against a non-existent channel on a live Integration Service connection. Maps to the [slack-api-error](../../../../../skills/uipath-troubleshoot/references/activity-packages/slack-activities/playbooks/slack-api-error.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, BusinessActivityExecutionException + Slack ProviderMessage) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the Slack error is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/0c745290f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/0c745290f5.json deleted file mode 100644 index b850825e2d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/0c745290f5.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/5274d54bd0.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/5274d54bd0.json deleted file mode 100644 index 075fddd719..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/5274d54bd0.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T11:08:44.768Z", - "Message": "Send Message to Channel: BusinessActivityExecutionException:\nError Code: [400]\nMessage: Bad Request - Often due to a missing request parameter\nRequestId: 6a43a3bce4b0cb1e6eda4378\nProviderMessage: {\"ok\":false,\"error\":\"channel_not_found\",\"warning\":\"missing_charset\",\"response_metadata\":{\"warnings\":[\"missing_charset\"]}}\n\r\nin Main.xaml\r\n at SendMessage \"Send Message to Channel\"\r\n at Sequence \"Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException\r\n at UiPath.BAF.Activities.BAFAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/7820c67204.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/7820c67204.json deleted file mode 100644 index 5aef93762e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/7820c67204.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "f0c8117e-38f1-474a-8c14-81f94c86816c", - "State": "Faulted", - "StartTime": "2026-06-20T11:08:39.840Z", - "EndTime": "2026-06-20T11:08:45.333Z", - "ReleaseName": "TeamNotifications", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50596529 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/README.md deleted file mode 100644 index 8cb9d08062..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Slack Send Message — channel_not_found (read the ProviderMessage, not the HTTP 400) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the `UiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException` case of `UiPath.Slack.IntegrationService.Activities.SendMessage`. - -## What this exercises - -A Slack **Send Message** ("Send Message to Channel") faults *after* its Integration Service connection resolves, because the Slack Web API rejects the call. The job ends Faulted with `BusinessActivityExecutionException` carrying `Error Code: [400]` and the raw Slack `ProviderMessage: {"ok":false,"error":"channel_not_found",...}`. The agent must read the **Slack `error` code** (`channel_not_found`) as the cause — not stop at the generic HTTP 400, and not blame connection resolution (that would be a `System.AggregateException` / `ConnectionException`, a different playbook). The fix is to correct the Channel argument. - -Signature captured verbatim from a real Orchestrator job produced by running a Slack `SendMessage` against a non-existent channel on a live Integration Service connection. Maps to the [slack-api-error](../../../../../skills/uipath-troubleshoot/references/activity-packages/slack-activities/playbooks/slack-api-error.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, BusinessActivityExecutionException + Slack ProviderMessage) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (no spans emitted) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the Slack error is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/a6c44081c4.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/a6c44081c4.json deleted file mode 100644 index e40a05dcfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/a6c44081c4.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f0c8117e-38f1-474a-8c14-81f94c86816c", - "StartTime": "2026-06-20T11:08:39.840Z", - "EndTime": "2026-06-20T11:08:45.333Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "30771724-17d3-4eb0-ba94-f1df799873e0", - "Info": "BusinessActivityExecutionException:\nError Code: [400]\nMessage: Bad Request - Often due to a missing request parameter\nRequestId: 6a43a3bce4b0cb1e6eda4378\nProviderMessage: {\"ok\":false,\"error\":\"channel_not_found\",\"warning\":\"missing_charset\",\"response_metadata\":{\"warnings\":[\"missing_charset\"]}}\n\r\n\r\nin Main.xaml\r\n at SendMessage \"Send Message to Channel\"\r\n at Sequence \"Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException: BusinessActivityExecutionException:\nError Code: [400]\nMessage: Bad Request - Often due to a missing request parameter\nRequestId: 6a43a3bce4b0cb1e6eda4378\nProviderMessage: {\"ok\":false,\"error\":\"channel_not_found\",\"warning\":\"missing_charset\",\"response_metadata\":{\"warnings\":[\"missing_charset\"]}}\n\r\n at UiPath.BAF.Activities.BAFAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T11:08:06.137Z", - "ReleaseName": "TeamNotifications", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108601, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T11:08:45.333Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f0c8117e38f1474a8c1481f94c86816c", - "Id": 50596529 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/manifest.json deleted file mode 100644 index b728938067..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of a faulted UiPath.Slack.IntegrationService.Activities SendMessage job. Canned uip CLI responses captured from a real Orchestrator job; PII scrubbed. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "0c745290f5.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e --state Faulted", - "file": "7820c67204.json" - }, - { - "match": "or jobs list --folder-key 2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", - "file": "7820c67204.json" - }, - { - "match": "or jobs list", - "file": "7820c67204.json" - }, - { - "match": "or jobs get f0c8117e-38f1-474a-8c14-81f94c86816c", - "file": "a6c44081c4.json" - }, - { - "match": "or jobs logs f0c8117e-38f1-474a-8c14-81f94c86816c --level Error", - "file": "5274d54bd0.json" - }, - { - "match": "or jobs logs f0c8117e-38f1-474a-8c14-81f94c86816c", - "file": "5274d54bd0.json" - }, - { - "match": "or jobs traces f0c8117e-38f1-474a-8c14-81f94c86816c", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key f0c8117e-38f1-474a-8c14-81f94c86816c", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get f0c8117e-38f1-474a-8c14-81f94c86816c", - "min": 1, - "description": "Agent fetches the faulted job to read its exception detail." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..0ae984b35c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0c8117e-38f1-474a-8c14-81f94c86816c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:39.840Z\",\r\n \"EndTime\": \"2026-06-20T11:08:45.333Z\",\r\n \"ReleaseName\": \"TeamNotifications\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50596529\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f0c8117e-38f1-474a-8c14-81f94c86816c", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T11:08:44.768Z\",\r\n \"Message\": \"Send Message to Channel: BusinessActivityExecutionException:\\nError Code: [400]\\nMessage: Bad Request - Often due to a missing request parameter\\nRequestId: 6a43a3bce4b0cb1e6eda4378\\nProviderMessage: {\\\"ok\\\":false,\\\"error\\\":\\\"channel_not_found\\\",\\\"warning\\\":\\\"missing_charset\\\",\\\"response_metadata\\\":{\\\"warnings\\\":[\\\"missing_charset\\\"]}}\\n\\r\\nin Main.xaml\\r\\n at SendMessage \\\"Send Message to Channel\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException\\r\\n at UiPath.BAF.Activities.BAFAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0c8117e-38f1-474a-8c14-81f94c86816c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:39.840Z\",\r\n \"EndTime\": \"2026-06-20T11:08:45.333Z\",\r\n \"ReleaseName\": \"TeamNotifications\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50596529\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "f0c8117e-38f1-474a-8c14-81f94c86816c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f0c8117e-38f1-474a-8c14-81f94c86816c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f0c8117e-38f1-474a-8c14-81f94c86816c\",\r\n \"StartTime\": \"2026-06-20T11:08:39.840Z\",\r\n \"EndTime\": \"2026-06-20T11:08:45.333Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"30771724-17d3-4eb0-ba94-f1df799873e0\",\r\n \"Info\": \"BusinessActivityExecutionException:\\nError Code: [400]\\nMessage: Bad Request - Often due to a missing request parameter\\nRequestId: 6a43a3bce4b0cb1e6eda4378\\nProviderMessage: {\\\"ok\\\":false,\\\"error\\\":\\\"channel_not_found\\\",\\\"warning\\\":\\\"missing_charset\\\",\\\"response_metadata\\\":{\\\"warnings\\\":[\\\"missing_charset\\\"]}}\\n\\r\\n\\r\\nin Main.xaml\\r\\n at SendMessage \\\"Send Message to Channel\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException: BusinessActivityExecutionException:\\nError Code: [400]\\nMessage: Bad Request - Often due to a missing request parameter\\nRequestId: 6a43a3bce4b0cb1e6eda4378\\nProviderMessage: {\\\"ok\\\":false,\\\"error\\\":\\\"channel_not_found\\\",\\\"warning\\\":\\\"missing_charset\\\",\\\"response_metadata\\\":{\\\"warnings\\\":[\\\"missing_charset\\\"]}}\\n\\r\\n at UiPath.BAF.Activities.BAFAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T11:08:06.137Z\",\r\n \"ReleaseName\": \"TeamNotifications\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108601,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T11:08:45.333Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f0c8117e38f1474a8c1481f94c86816c\",\r\n \"Id\": 50596529\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f0c8117e-38f1-474a-8c14-81f94c86816c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T11:08:44.768Z\",\r\n \"Message\": \"Send Message to Channel: BusinessActivityExecutionException:\\nError Code: [400]\\nMessage: Bad Request - Often due to a missing request parameter\\nRequestId: 6a43a3bce4b0cb1e6eda4378\\nProviderMessage: {\\\"ok\\\":false,\\\"error\\\":\\\"channel_not_found\\\",\\\"warning\\\":\\\"missing_charset\\\",\\\"response_metadata\\\":{\\\"warnings\\\":[\\\"missing_charset\\\"]}}\\n\\r\\nin Main.xaml\\r\\n at SendMessage \\\"Send Message to Channel\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.BAF.Infrastructure.Exceptions.BusinessActivityExecutionException\\r\\n at UiPath.BAF.Activities.BAFAsyncCodeActivity`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "f0c8117e-38f1-474a-8c14-81f94c86816c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2b3c4d5e-6f7a-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0c8117e-38f1-474a-8c14-81f94c86816c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:39.840Z\",\r\n \"EndTime\": \"2026-06-20T11:08:45.333Z\",\r\n \"ReleaseName\": \"TeamNotifications\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50596529\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get f0c8117e-38f1-474a-8c14-81f94c86816c", + "min": 1, + "description": "Agent fetches the faulted job to read its exception detail." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/task.yaml index 3a452caac8..2a5497f2f5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/slack-send-message-channel-not-found/task.yaml @@ -20,20 +20,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/README.md new file mode 100644 index 0000000000..b8fe58b650 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/README.md @@ -0,0 +1,36 @@ +# Compress/Extract Files Failure — Corrupt Archive + +This scenario replays a real faulted Orchestrator job where the +`Extract/Unzip Files` activity cannot open its input ZIP. Orchestrator +surfaces a `CompressionException: Cannot extract data from archive`, and +the inner exception (`ICSharpCode.SharpZipLib.Zip.ZipException: Cannot +find central directory`) proves the file is not a valid, complete ZIP. + +## What this scenario uncovers + +**Root Cause:** The file at +`C:\AutomationData\Invoices\batch_2026.zip` is not a valid/complete ZIP — +its central-directory record is missing. `Extract/Unzip Files` cannot +open it, so the `InvoiceIntake` job faults. + +This maps to: +`references/activity-packages/system-activities/playbooks/compress-extract-files-failed.md` +(the "Cannot extract data from archive" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Specifically, the agent must name the archive as corrupt/incomplete (missing central directory), not a permissions or missing-file problem + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/29cb52f76a.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/29cb52f76a.json deleted file mode 100644 index 9dd3ab2452..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/29cb52f76a.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T09:15:13.180Z", - "Message": "Extract/Unzip Files: Cannot extract data from archive \"C:\\AutomationData\\Invoices\\batch_2026.zip\"." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-24T09:15:13.210Z", - "Message": "Cannot extract data from archive \"C:\\AutomationData\\Invoices\\batch_2026.zip\".\r\nin Main.xaml\r\n at ExtractFiles \"Extract/Unzip Files\"\r\n at Sequence \"Main Sequence\"\r\n at InvoiceIntake \"InvoiceIntake\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/8dc1b36d84.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/8dc1b36d84.json deleted file mode 100644 index e86de4a798..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/8dc1b36d84.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c", - "StartTime": "2026-06-24T09:15:02.010Z", - "EndTime": "2026-06-24T09:15:13.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9d3f5a7b-2c4e-4f6a-8b0c-1d2e3f4a5b6c", - "Info": "Extraction Failed\r\n\r\nCheck if the archive is corrupted or if you have write permissions to the destination\r\n\r\nin Main.xaml\r\n at ExtractFiles \"Extract/Unzip Files\"\r\n at Sequence \"Main Sequence\"\r\n at InvoiceIntake \"InvoiceIntake\"\r\n\r\nUiPath.Activities.System.Compression.CompressionException: Cannot extract data from archive \"C:\\AutomationData\\Invoices\\batch_2026.zip\".\r\n ---> ICSharpCode.SharpZipLib.Zip.ZipException: Cannot find central directory\r\n at ICSharpCode.SharpZipLib.Zip.ZipFile.ReadEntries()\r\n at UiPath.Activities.System.Compression.Zip.ZipExtractor.ExtractAsync(CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Activities.System.Compression.Workflow.ExtractFiles.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)", - "CreationTime": "2026-06-24T09:15:01.740Z", - "ReleaseName": "InvoiceIntake", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5102911, - "OrganizationUnitId": 7791618, - "FolderKey": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOT", - "OrchestratorUserIdentity": "svc.finance", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-24T09:15:13.220Z", - "JobError": {}, - "ErrorCode": "System.Compression.Sys.ExtractFailed", - "TraceId": "3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c", - "Id": 671104220 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/README.md deleted file mode 100644 index bc8985788f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Compress/Extract Files Failure — Corrupt Archive - -This scenario replays a real faulted Orchestrator job where the -`Extract/Unzip Files` activity cannot open its input ZIP. Orchestrator -surfaces a `CompressionException: Cannot extract data from archive`, and -the inner exception (`ICSharpCode.SharpZipLib.Zip.ZipException: Cannot -find central directory`) proves the file is not a valid, complete ZIP. - -## What this scenario uncovers - -**Root Cause:** The file at -`C:\AutomationData\Invoices\batch_2026.zip` is not a valid/complete ZIP — -its central-directory record is missing. `Extract/Unzip Files` cannot -open it, so the `InvoiceIntake` job faults. - -This maps to: -`references/activity-packages/system-activities/playbooks/compress-extract-files-failed.md` -(the "Cannot extract data from archive" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | -| `fixtures/mocks/responses/*.json` | canned `uip` responses replaying the captured job/logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Specifically, the agent must name the archive as corrupt/incomplete (missing central directory), not a permissions or missing-file problem diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/bd6f752385.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/bd6f752385.json deleted file mode 100644 index 5580d2b94d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/bd6f752385.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c", - "State": "Faulted", - "ProcessName": "InvoiceIntake", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-24T09:15:02.010Z", - "EndTime": "2026-06-24T09:15:13.220Z", - "CreationTime": "2026-06-24T09:15:01.740Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/e282487c93.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/e282487c93.json deleted file mode 100644 index 871157f284..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/e282487c93.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Name": "Finance", - "Path": "Finance", - "Description": "Finance department automation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/manifest.json deleted file mode 100644 index dd37e26a4e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/m/r/manifest.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-replay fixtures for a captured InvoiceIntake faulted job (Extract/Unzip Files \u2014 corrupt archive). Names/paths/keys/timestamps scrubbed; exception text + activity class preserved verbatim.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e282487c93.json" - }, - { - "match": "or jobs list --folder-key \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\" --state Faulted --output json", - "file": "bd6f752385.json" - }, - { - "match": "or jobs list --folder-key 4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f --state Faulted --output json", - "file": "bd6f752385.json" - }, - { - "match": "or jobs list --process-name InvoiceIntake", - "file": "bd6f752385.json" - }, - { - "match": "or jobs list --folder-path Finance", - "file": "bd6f752385.json" - }, - { - "match": "or jobs list --folder-path \"Finance\"", - "file": "bd6f752385.json" - }, - { - "match": "or jobs get \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\" --output json", - "file": "8dc1b36d84.json" - }, - { - "match": "or jobs get 3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c --output json", - "file": "8dc1b36d84.json" - }, - { - "match": "or jobs logs \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\" --level Error --output json", - "file": "29cb52f76a.json" - }, - { - "match": "or jobs logs 3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c --level Error --output json", - "file": "29cb52f76a.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/uip-fixture.json new file mode 100644 index 0000000000..3d2a7d9689 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/data/uip-fixture.json @@ -0,0 +1,89 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceIntake\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T09:15:02.010Z\",\r\n \"EndTime\": \"2026-06-24T09:15:13.220Z\",\r\n \"CreationTime\": \"2026-06-24T09:15:01.740Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T09:15:13.180Z\",\r\n \"Message\": \"Extract/Unzip Files: Cannot extract data from archive \\\"C:\\\\AutomationData\\\\Invoices\\\\batch_2026.zip\\\".\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T09:15:13.210Z\",\r\n \"Message\": \"Cannot extract data from archive \\\"C:\\\\AutomationData\\\\Invoices\\\\batch_2026.zip\\\".\\r\\nin Main.xaml\\r\\n at ExtractFiles \\\"Extract/Unzip Files\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at InvoiceIntake \\\"InvoiceIntake\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "InvoiceIntake" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceIntake\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T09:15:02.010Z\",\r\n \"EndTime\": \"2026-06-24T09:15:13.220Z\",\r\n \"CreationTime\": \"2026-06-24T09:15:01.740Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Finance" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceIntake\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T09:15:02.010Z\",\r\n \"EndTime\": \"2026-06-24T09:15:13.220Z\",\r\n \"CreationTime\": \"2026-06-24T09:15:01.740Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\",\r\n \"StartTime\": \"2026-06-24T09:15:02.010Z\",\r\n \"EndTime\": \"2026-06-24T09:15:13.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9d3f5a7b-2c4e-4f6a-8b0c-1d2e3f4a5b6c\",\r\n \"Info\": \"Extraction Failed\\r\\n\\r\\nCheck if the archive is corrupted or if you have write permissions to the destination\\r\\n\\r\\nin Main.xaml\\r\\n at ExtractFiles \\\"Extract/Unzip Files\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at InvoiceIntake \\\"InvoiceIntake\\\"\\r\\n\\r\\nUiPath.Activities.System.Compression.CompressionException: Cannot extract data from archive \\\"C:\\\\AutomationData\\\\Invoices\\\\batch_2026.zip\\\".\\r\\n ---> ICSharpCode.SharpZipLib.Zip.ZipException: Cannot find central directory\\r\\n at ICSharpCode.SharpZipLib.Zip.ZipFile.ReadEntries()\\r\\n at UiPath.Activities.System.Compression.Zip.ZipExtractor.ExtractAsync(CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Activities.System.Compression.Workflow.ExtractFiles.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\",\r\n \"CreationTime\": \"2026-06-24T09:15:01.740Z\",\r\n \"ReleaseName\": \"InvoiceIntake\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5102911,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOT\",\r\n \"OrchestratorUserIdentity\": \"svc.finance\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-24T09:15:13.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"System.Compression.Sys.ExtractFailed\",\r\n \"TraceId\": \"3e9a1c74-5b2d-4e8f-a1c6-7d9e0f2a3b4c\",\r\n \"Id\": 671104220\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"Finance department automation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/task.yaml index 7a40539afb..5dc2bc2912 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-compress-extract-corrupt-archive/task.yaml @@ -22,21 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/README.md new file mode 100644 index 0000000000..30005ce8fd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/README.md @@ -0,0 +1,37 @@ +# Copy File Failure — Destination Already Exists + +This scenario replays a real faulted Orchestrator job where the +`Copy File` activity targets a destination path that already exists and +its Overwrite option is off. The modern file activity surfaces the raw +`System.IO.IOException: The file '...' already exists.` instead of a +UiPath `FileSystemException`. + +## What this scenario uncovers + +**Root Cause:** `Copy File` writes to +`C:\AutomationData\Ledger\archive\ledger_2026.txt`, which already exists, +and Overwrite is off. `System.IO.File.Copy` throws +`IOException: ... already exists.` The failure is a destination conflict, +not a missing source or a permissions problem. + +This maps to: +`references/activity-packages/system-activities/playbooks/file-folder-operation-failed.md` +(the "Destination already exists" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Specifically, the agent must name the destination-exists + Overwrite-off conflict, not a source-missing or permissions failure + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/8ff2e6e5b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/8ff2e6e5b6.json deleted file mode 100644 index eb69790568..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/8ff2e6e5b6.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T11:20:07.060Z", - "Message": "Copy File: The file 'C:\\AutomationData\\Ledger\\archive\\ledger_2026.txt' already exists." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-24T11:20:07.090Z", - "Message": "The file 'C:\\AutomationData\\Ledger\\archive\\ledger_2026.txt' already exists.\r\nin Main.xaml\r\n at CopyFile \"Copy File\"\r\n at Sequence \"Main Sequence\"\r\n at LedgerConsolidation \"LedgerConsolidation\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/README.md deleted file mode 100644 index a2973cd312..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Copy File Failure — Destination Already Exists - -This scenario replays a real faulted Orchestrator job where the -`Copy File` activity targets a destination path that already exists and -its Overwrite option is off. The modern file activity surfaces the raw -`System.IO.IOException: The file '...' already exists.` instead of a -UiPath `FileSystemException`. - -## What this scenario uncovers - -**Root Cause:** `Copy File` writes to -`C:\AutomationData\Ledger\archive\ledger_2026.txt`, which already exists, -and Overwrite is off. `System.IO.File.Copy` throws -`IOException: ... already exists.` The failure is a destination conflict, -not a missing source or a permissions problem. - -This maps to: -`references/activity-packages/system-activities/playbooks/file-folder-operation-failed.md` -(the "Destination already exists" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | -| `fixtures/mocks/responses/*.json` | canned `uip` responses replaying the captured job/logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Specifically, the agent must name the destination-exists + Overwrite-off conflict, not a source-missing or permissions failure diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/e282487c93.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/e282487c93.json deleted file mode 100644 index 871157f284..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/e282487c93.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Name": "Finance", - "Path": "Finance", - "Description": "Finance department automation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/eb29e7ced3.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/eb29e7ced3.json deleted file mode 100644 index 2fc8f90540..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/eb29e7ced3.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71", - "State": "Faulted", - "ProcessName": "LedgerConsolidation", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-24T11:20:05.900Z", - "EndTime": "2026-06-24T11:20:07.100Z", - "CreationTime": "2026-06-24T11:20:05.640Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/fcbf8f953c.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/fcbf8f953c.json deleted file mode 100644 index 6c38579363..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/fcbf8f953c.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71", - "StartTime": "2026-06-24T11:20:05.900Z", - "EndTime": "2026-06-24T11:20:07.100Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "3c5e7a9b-4d6f-4a8b-9c0d-1e2f3a4b5c6d", - "Info": "The file 'C:\\AutomationData\\Ledger\\archive\\ledger_2026.txt' already exists.\r\n\r\nin Main.xaml\r\n at CopyFile \"Copy File\"\r\n at Sequence \"Main Sequence\"\r\n at LedgerConsolidation \"LedgerConsolidation\"\r\n\r\nSystem.IO.IOException: The file 'C:\\AutomationData\\Ledger\\archive\\ledger_2026.txt' already exists.\r\n at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)\r\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)", - "CreationTime": "2026-06-24T11:20:05.640Z", - "ReleaseName": "LedgerConsolidation", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5102913, - "OrganizationUnitId": 7791618, - "FolderKey": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOT", - "OrchestratorUserIdentity": "svc.finance", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-24T11:20:07.100Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71", - "Id": 671104222 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/manifest.json deleted file mode 100644 index 7828f57689..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/m/r/manifest.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-replay fixtures for a captured LedgerConsolidation faulted job (Copy File \u2014 destination already exists). Names/paths/keys/timestamps scrubbed; exception text + activity class preserved verbatim.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e282487c93.json" - }, - { - "match": "or jobs list --folder-key \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\" --state Faulted --output json", - "file": "eb29e7ced3.json" - }, - { - "match": "or jobs list --folder-key 4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f --state Faulted --output json", - "file": "eb29e7ced3.json" - }, - { - "match": "or jobs list --process-name LedgerConsolidation", - "file": "eb29e7ced3.json" - }, - { - "match": "or jobs list --folder-path Finance", - "file": "eb29e7ced3.json" - }, - { - "match": "or jobs list --folder-path \"Finance\"", - "file": "eb29e7ced3.json" - }, - { - "match": "or jobs get \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\" --output json", - "file": "fcbf8f953c.json" - }, - { - "match": "or jobs get c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71 --output json", - "file": "fcbf8f953c.json" - }, - { - "match": "or jobs logs \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\" --level Error --output json", - "file": "8ff2e6e5b6.json" - }, - { - "match": "or jobs logs c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71 --level Error --output json", - "file": "8ff2e6e5b6.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/uip-fixture.json new file mode 100644 index 0000000000..4fc95170f3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/data/uip-fixture.json @@ -0,0 +1,89 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LedgerConsolidation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T11:20:05.900Z\",\r\n \"EndTime\": \"2026-06-24T11:20:07.100Z\",\r\n \"CreationTime\": \"2026-06-24T11:20:05.640Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T11:20:07.060Z\",\r\n \"Message\": \"Copy File: The file 'C:\\\\AutomationData\\\\Ledger\\\\archive\\\\ledger_2026.txt' already exists.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T11:20:07.090Z\",\r\n \"Message\": \"The file 'C:\\\\AutomationData\\\\Ledger\\\\archive\\\\ledger_2026.txt' already exists.\\r\\nin Main.xaml\\r\\n at CopyFile \\\"Copy File\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at LedgerConsolidation \\\"LedgerConsolidation\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "LedgerConsolidation" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LedgerConsolidation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T11:20:05.900Z\",\r\n \"EndTime\": \"2026-06-24T11:20:07.100Z\",\r\n \"CreationTime\": \"2026-06-24T11:20:05.640Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Finance" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LedgerConsolidation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T11:20:05.900Z\",\r\n \"EndTime\": \"2026-06-24T11:20:07.100Z\",\r\n \"CreationTime\": \"2026-06-24T11:20:05.640Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\",\r\n \"StartTime\": \"2026-06-24T11:20:05.900Z\",\r\n \"EndTime\": \"2026-06-24T11:20:07.100Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"3c5e7a9b-4d6f-4a8b-9c0d-1e2f3a4b5c6d\",\r\n \"Info\": \"The file 'C:\\\\AutomationData\\\\Ledger\\\\archive\\\\ledger_2026.txt' already exists.\\r\\n\\r\\nin Main.xaml\\r\\n at CopyFile \\\"Copy File\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at LedgerConsolidation \\\"LedgerConsolidation\\\"\\r\\n\\r\\nSystem.IO.IOException: The file 'C:\\\\AutomationData\\\\Ledger\\\\archive\\\\ledger_2026.txt' already exists.\\r\\n at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)\\r\\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\",\r\n \"CreationTime\": \"2026-06-24T11:20:05.640Z\",\r\n \"ReleaseName\": \"LedgerConsolidation\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5102913,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOT\",\r\n \"OrchestratorUserIdentity\": \"svc.finance\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-24T11:20:07.100Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c7e0a2b4-9d1f-4e63-8a5c-2b3d4e5f6a71\",\r\n \"Id\": 671104222\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"Finance department automation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/task.yaml index bd6954722a..5e1b5d2456 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-copy-file-destination-exists/task.yaml @@ -23,21 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/README.md new file mode 100644 index 0000000000..9b203ebdb7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/README.md @@ -0,0 +1,37 @@ +# Download File Failure — DNS / Transport Failure + +This scenario replays a real faulted Orchestrator job where the +`Download File from URL` activity fails at the transport layer. The job +throws `System.Net.Http.HttpRequestException` wrapping a +`System.Net.Sockets.SocketException` ("The requested name is valid, but +no data of the requested type was found") — a DNS / name-resolution +failure for the target host. + +## What this scenario uncovers + +**Root Cause:** The download host +`files.acme-vendorportal.example.com` does not resolve from the robot. +The failure is at the network/transport layer (DNS), not a missing file, +HTTP 404, or authentication error. + +This maps to: +`references/activity-packages/system-activities/playbooks/download-file-failed.md` +(the "Transport failure — HttpRequestException" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Specifically, the agent must name a DNS / name-resolution / transport failure for the host, not a file-not-found or auth failure + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/4b06f34d03.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/4b06f34d03.json deleted file mode 100644 index a74adf1b6b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/4b06f34d03.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34", - "State": "Faulted", - "ProcessName": "VendorPortalSync", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-24T10:02:31.400Z", - "EndTime": "2026-06-24T10:02:33.010Z", - "CreationTime": "2026-06-24T10:02:31.120Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/945dc30a46.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/945dc30a46.json deleted file mode 100644 index e6f97d009c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/945dc30a46.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34", - "StartTime": "2026-06-24T10:02:31.400Z", - "EndTime": "2026-06-24T10:02:33.010Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Info": "The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\r\n\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at VendorPortalSync \"VendorPortalSync\"\r\n\r\nSystem.Net.Http.HttpRequestException: The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\r\n ---> System.Net.Sockets.SocketException: The requested name is valid, but no data of the requested type was found.\r\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Core.Activities.TaskAsyncCodeActivityWithResult`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)", - "CreationTime": "2026-06-24T10:02:31.120Z", - "ReleaseName": "VendorPortalSync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5102912, - "OrganizationUnitId": 7791618, - "FolderKey": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOT", - "OrchestratorUserIdentity": "svc.finance", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-24T10:02:33.010Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34", - "Id": 671104221 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/96eb565046.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/96eb565046.json deleted file mode 100644 index 9e47887bab..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/96eb565046.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T10:02:32.970Z", - "Message": "Download File from URL: The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-24T10:02:33.000Z", - "Message": "The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\r\nin Main.xaml\r\n at DownloadFileFromUrl \"Download File from URL\"\r\n at Sequence \"Main Sequence\"\r\n at VendorPortalSync \"VendorPortalSync\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/README.md deleted file mode 100644 index a038baa07b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Download File Failure — DNS / Transport Failure - -This scenario replays a real faulted Orchestrator job where the -`Download File from URL` activity fails at the transport layer. The job -throws `System.Net.Http.HttpRequestException` wrapping a -`System.Net.Sockets.SocketException` ("The requested name is valid, but -no data of the requested type was found") — a DNS / name-resolution -failure for the target host. - -## What this scenario uncovers - -**Root Cause:** The download host -`files.acme-vendorportal.example.com` does not resolve from the robot. -The failure is at the network/transport layer (DNS), not a missing file, -HTTP 404, or authentication error. - -This maps to: -`references/activity-packages/system-activities/playbooks/download-file-failed.md` -(the "Transport failure — HttpRequestException" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | -| `fixtures/mocks/responses/*.json` | canned `uip` responses replaying the captured job/logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Specifically, the agent must name a DNS / name-resolution / transport failure for the host, not a file-not-found or auth failure diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/e282487c93.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/e282487c93.json deleted file mode 100644 index 871157f284..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/e282487c93.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Name": "Finance", - "Path": "Finance", - "Description": "Finance department automation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/manifest.json deleted file mode 100644 index 7f160696fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/m/r/manifest.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-replay fixtures for a captured VendorPortalSync faulted job (Download File from URL \u2014 DNS/transport failure). Names/paths/keys/timestamps scrubbed; exception text + activity class preserved verbatim.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e282487c93.json" - }, - { - "match": "or jobs list --folder-key \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\" --state Faulted --output json", - "file": "4b06f34d03.json" - }, - { - "match": "or jobs list --folder-key 4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f --state Faulted --output json", - "file": "4b06f34d03.json" - }, - { - "match": "or jobs list --process-name VendorPortalSync", - "file": "4b06f34d03.json" - }, - { - "match": "or jobs list --folder-path Finance", - "file": "4b06f34d03.json" - }, - { - "match": "or jobs list --folder-path \"Finance\"", - "file": "4b06f34d03.json" - }, - { - "match": "or jobs get \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\" --output json", - "file": "945dc30a46.json" - }, - { - "match": "or jobs get 8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34 --output json", - "file": "945dc30a46.json" - }, - { - "match": "or jobs logs \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\" --level Error --output json", - "file": "96eb565046.json" - }, - { - "match": "or jobs logs 8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34 --level Error --output json", - "file": "96eb565046.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/uip-fixture.json new file mode 100644 index 0000000000..25af05ea93 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/data/uip-fixture.json @@ -0,0 +1,89 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VendorPortalSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T10:02:31.400Z\",\r\n \"EndTime\": \"2026-06-24T10:02:33.010Z\",\r\n \"CreationTime\": \"2026-06-24T10:02:31.120Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T10:02:32.970Z\",\r\n \"Message\": \"Download File from URL: The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T10:02:33.000Z\",\r\n \"Message\": \"The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at VendorPortalSync \\\"VendorPortalSync\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "VendorPortalSync" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VendorPortalSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T10:02:31.400Z\",\r\n \"EndTime\": \"2026-06-24T10:02:33.010Z\",\r\n \"CreationTime\": \"2026-06-24T10:02:31.120Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Finance" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VendorPortalSync\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T10:02:31.400Z\",\r\n \"EndTime\": \"2026-06-24T10:02:33.010Z\",\r\n \"CreationTime\": \"2026-06-24T10:02:31.120Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\",\r\n \"StartTime\": \"2026-06-24T10:02:31.400Z\",\r\n \"EndTime\": \"2026-06-24T10:02:33.010Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Info\": \"The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\\r\\n\\r\\nin Main.xaml\\r\\n at DownloadFileFromUrl \\\"Download File from URL\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at VendorPortalSync \\\"VendorPortalSync\\\"\\r\\n\\r\\nSystem.Net.Http.HttpRequestException: The requested name is valid, but no data of the requested type was found. (files.acme-vendorportal.example.com:443)\\r\\n ---> System.Net.Sockets.SocketException: The requested name is valid, but no data of the requested type was found.\\r\\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Core.Activities.TaskAsyncCodeActivityWithResult`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\",\r\n \"CreationTime\": \"2026-06-24T10:02:31.120Z\",\r\n \"ReleaseName\": \"VendorPortalSync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5102912,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOT\",\r\n \"OrchestratorUserIdentity\": \"svc.finance\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-24T10:02:33.010Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8b2d5f61-4c3e-4a9d-b7e2-1f6a8c0d9e34\",\r\n \"Id\": 671104221\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"Finance department automation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/task.yaml index 1191815b65..b49481a1ca 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-download-file-dns-failure/task.yaml @@ -23,21 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/README.md new file mode 100644 index 0000000000..3e2879fca9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/README.md @@ -0,0 +1,37 @@ +# Get Queue Item Failure — Queue Does Not Exist + +This scenario replays a real faulted Orchestrator job where the +`Get Queue Item` activity references a queue that does not exist in the +folder where the job runs. Orchestrator returns HTTP 404 / error code +1002 (`OrchestratorHttpException`), surfaced under the generic "HTTP +Error" job header. + +## What this scenario uncovers + +**Root Cause:** The queue `OrdersToProcess` does not exist in the +`Finance` folder. `Get Queue Item` calls Orchestrator and gets HTTP 404 +/ error code 1002 (`OrdersToProcess does not exist`). Despite the generic +"HTTP Error / Check HTTP connectivity" header, the response body names +the missing queue — this is not a connectivity problem. + +This maps to: +`references/activity-packages/system-activities/playbooks/queue-transaction-activity-failed.md` +(the 404 / "does not exist. Error code: 1002" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Specifically, the agent must name the missing queue `OrdersToProcess` in the Finance folder (404 / code 1002), NOT a generic HTTP connectivity failure + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/5afb9ccbf4.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/5afb9ccbf4.json deleted file mode 100644 index a273aee2e7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/5afb9ccbf4.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82", - "State": "Faulted", - "ProcessName": "OrderProcessor", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-24T12:45:10.300Z", - "EndTime": "2026-06-24T12:45:11.700Z", - "CreationTime": "2026-06-24T12:45:10.020Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/README.md deleted file mode 100644 index e4ee5def6f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Get Queue Item Failure — Queue Does Not Exist - -This scenario replays a real faulted Orchestrator job where the -`Get Queue Item` activity references a queue that does not exist in the -folder where the job runs. Orchestrator returns HTTP 404 / error code -1002 (`OrchestratorHttpException`), surfaced under the generic "HTTP -Error" job header. - -## What this scenario uncovers - -**Root Cause:** The queue `OrdersToProcess` does not exist in the -`Finance` folder. `Get Queue Item` calls Orchestrator and gets HTTP 404 -/ error code 1002 (`OrdersToProcess does not exist`). Despite the generic -"HTTP Error / Check HTTP connectivity" header, the response body names -the missing queue — this is not a connectivity problem. - -This maps to: -`references/activity-packages/system-activities/playbooks/queue-transaction-activity-failed.md` -(the 404 / "does not exist. Error code: 1002" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json`) | -| `fixtures/mocks/responses/*.json` | canned `uip` responses replaying the captured job/logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Specifically, the agent must name the missing queue `OrdersToProcess` in the Finance folder (404 / code 1002), NOT a generic HTTP connectivity failure diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/a9a84840db.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/a9a84840db.json deleted file mode 100644 index 9accf8995f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/a9a84840db.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82", - "StartTime": "2026-06-24T12:45:10.300Z", - "EndTime": "2026-06-24T12:45:11.700Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5e7a9b1c-4d6f-4a8b-9c0d-1e2f3a4b5c6d", - "Info": "HTTP Error\r\n\r\nCheck HTTP connectivity and verify the response error details\r\n\r\nin Main.xaml\r\n at GetQueueItem \"Get Queue Item\"\r\n at Sequence \"Main Sequence\"\r\n at OrderProcessor \"OrderProcessor\"\r\n\r\nUiPath.Core.Activities.OrchestratorHttpException: Status code: 404 (Not Found). Orchestrator response: \r\nOrdersToProcess does not exist. Error code: 1002\r\n at UiPath.Core.Activities.GetQueueItem.Execute(CodeActivityContext context)", - "CreationTime": "2026-06-24T12:45:10.020Z", - "ReleaseName": "OrderProcessor", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5102914, - "OrganizationUnitId": 7791618, - "FolderKey": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOT", - "OrchestratorUserIdentity": "svc.finance", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-24T12:45:11.700Z", - "JobError": {}, - "ErrorCode": "System.Utilities.Sys.HttpError", - "TraceId": "d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82", - "Id": 671104223 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/d0ae845f72.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/d0ae845f72.json deleted file mode 100644 index e52acf8ae1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/d0ae845f72.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T12:45:11.660Z", - "Message": "Get Queue Item: Status code: 404 (Not Found). Orchestrator response: \r\nOrdersToProcess does not exist. Error code: 1002" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-24T12:45:11.690Z", - "Message": "Status code: 404 (Not Found). Orchestrator response: \r\nOrdersToProcess does not exist. Error code: 1002\r\nin Main.xaml\r\n at GetQueueItem \"Get Queue Item\"\r\n at Sequence \"Main Sequence\"\r\n at OrderProcessor \"OrderProcessor\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/e282487c93.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/e282487c93.json deleted file mode 100644 index 871157f284..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/e282487c93.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", - "Name": "Finance", - "Path": "Finance", - "Description": "Finance department automation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/manifest.json deleted file mode 100644 index a6aa5cf99c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-replay fixtures for a captured OrderProcessor faulted job (Get Queue Item \u2014 queue does not exist, HTTP 404 / code 1002). Names/paths/keys/timestamps scrubbed; exception text + activity class preserved verbatim.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e282487c93.json" - }, - { - "match": "or jobs list --folder-key \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\" --state Faulted --output json", - "file": "5afb9ccbf4.json" - }, - { - "match": "or jobs list --folder-key 4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f --state Faulted --output json", - "file": "5afb9ccbf4.json" - }, - { - "match": "or jobs list --process-name OrderProcessor", - "file": "5afb9ccbf4.json" - }, - { - "match": "or jobs list --folder-path Finance", - "file": "5afb9ccbf4.json" - }, - { - "match": "or jobs list --folder-path \"Finance\"", - "file": "5afb9ccbf4.json" - }, - { - "match": "or jobs get \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\" --output json", - "file": "a9a84840db.json" - }, - { - "match": "or jobs get d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82 --output json", - "file": "a9a84840db.json" - }, - { - "match": "or jobs logs \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\" --level Error --output json", - "file": "d0ae845f72.json" - }, - { - "match": "or jobs logs d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82 --level Error --output json", - "file": "d0ae845f72.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..0fd74d3876 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/data/uip-fixture.json @@ -0,0 +1,89 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderProcessor\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T12:45:10.300Z\",\r\n \"EndTime\": \"2026-06-24T12:45:11.700Z\",\r\n \"CreationTime\": \"2026-06-24T12:45:10.020Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T12:45:11.660Z\",\r\n \"Message\": \"Get Queue Item: Status code: 404 (Not Found). Orchestrator response: \\r\\nOrdersToProcess does not exist. Error code: 1002\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T12:45:11.690Z\",\r\n \"Message\": \"Status code: 404 (Not Found). Orchestrator response: \\r\\nOrdersToProcess does not exist. Error code: 1002\\r\\nin Main.xaml\\r\\n at GetQueueItem \\\"Get Queue Item\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at OrderProcessor \\\"OrderProcessor\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "OrderProcessor" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderProcessor\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T12:45:10.300Z\",\r\n \"EndTime\": \"2026-06-24T12:45:11.700Z\",\r\n \"CreationTime\": \"2026-06-24T12:45:10.020Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Finance" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderProcessor\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-24T12:45:10.300Z\",\r\n \"EndTime\": \"2026-06-24T12:45:11.700Z\",\r\n \"CreationTime\": \"2026-06-24T12:45:10.020Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\",\r\n \"StartTime\": \"2026-06-24T12:45:10.300Z\",\r\n \"EndTime\": \"2026-06-24T12:45:11.700Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5e7a9b1c-4d6f-4a8b-9c0d-1e2f3a4b5c6d\",\r\n \"Info\": \"HTTP Error\\r\\n\\r\\nCheck HTTP connectivity and verify the response error details\\r\\n\\r\\nin Main.xaml\\r\\n at GetQueueItem \\\"Get Queue Item\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at OrderProcessor \\\"OrderProcessor\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorHttpException: Status code: 404 (Not Found). Orchestrator response: \\r\\nOrdersToProcess does not exist. Error code: 1002\\r\\n at UiPath.Core.Activities.GetQueueItem.Execute(CodeActivityContext context)\",\r\n \"CreationTime\": \"2026-06-24T12:45:10.020Z\",\r\n \"ReleaseName\": \"OrderProcessor\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5102914,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOT\",\r\n \"OrchestratorUserIdentity\": \"svc.finance\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-24T12:45:11.700Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"System.Utilities.Sys.HttpError\",\r\n \"TraceId\": \"d1f3b5c7-2e46-4a8b-9c0d-3e4f5a6b7c82\",\r\n \"Id\": 671104223\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b1c7e90-2a3d-4f5e-9c8b-1a2b3c4d5e6f\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"Finance department automation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/task.yaml index cd69cc4b75..8337fe41a7 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-get-queue-item-not-found/task.yaml @@ -24,21 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/README.md new file mode 100644 index 0000000000..4277ce4a03 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/README.md @@ -0,0 +1,87 @@ +# Get Asset Failure — Activity Bug / Silent Failure (UiPath.System.Activities 22.10.x) + +This scenario reproduces a runtime `Get Credential` failure that is +**silent** at the activity layer — the activity itself completes +without throwing an exception, but its output variables remain `null`. +The downstream `LogMessage` activity then throws +`NullReferenceException` consuming the null outputs. The job logs do +NOT show any error from `Get Credential` itself — only the downstream +NRE. + +This is the documented `UiPath.System.Activities` **22.10.x** bug +where variables bound via `Ctrl+K` in the activity's property grid +do not receive output values at runtime. + +## What this scenario uncovers + +**Root Cause:** The project targets `UiPath.System.Activities: [22.10.5]`. +That version has a documented bug where the `Get Credential`, +`Get Asset`, and `Get Orchestrator Asset` activities silently fail to +populate output variables when those variables were created via +`Ctrl+K` during activity configuration. The fix is to upgrade the +package, or pre-create the output variables in the Variables panel +BEFORE wiring them into the activity. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-activity-bug-silent-failure.md` +(medium-confidence playbook). + +> **Why this is harder than other GetAsset scenarios:** every other +> playbook in the family is keyed off a distinctive runtime error +> from the Get Credential / Get Asset activity itself. This one is +> keyed off the **absence** of such an error combined with a +> downstream null-consumer fault AND the package version. The agent +> has to (a) trace backward from the downstream NRE to find the +> silent activity, and (b) inspect `project.json` to connect the +> pattern to the package bug. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project pinned to `UiPath.System.Activities: [22.10.5]` | + +The decisive evidence: + +1. **Job logs** show `NullReferenceException` at `LogMessage "Log Message"` — NOT at `GetRobotCredential "Get Credential"`. There is no error log entry naming Get Credential. +2. **`project.json`** lists `UiPath.System.Activities: [22.10.5]` — the version with the documented Ctrl+K output-binding bug. +3. **Asset list** shows `myHiddenAsset` is present, correctly typed, with `ValueScope: "Global"` — the asset and folder are fine; the failure isn't at the asset layer. +4. **`users list`** shows `RobotUser1` has the right roles and a valid license — rules out permission / authentication / licensing issues. + +## How this differs from sibling scenarios + +| Dimension | Other GetAsset scenarios | `activity-silent-failure` (this) | +|---|---|---| +| Get Credential activity throws an error log entry? | yes | **no — completes silently** | +| Where the error log entry comes from | the Get Credential activity itself | a **downstream** activity (LogMessage) | +| Decisive evidence in `project.json`? | no | **yes — package version 22.10.x** | +| Asset layer healthy? | varies | **yes** | +| Robot / role / license healthy? | varies | **yes** | +| Matched playbook | one of get-asset-{not-found, folder-scope-mismatch, permission-denied, wrong-activity-type, per-robot-no-value, robot-not-authenticated, external-vault-failure} | `get-asset-activity-bug-silent-failure.md` | + +This is the only scenario in the family where the agent has to +troubleshoot **backward** from a non-asset-shaped error (NRE at +LogMessage) AND inspect `project.json` to find a package-version +anchor. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) recognize that Get Credential completed silently, (b) name the package version `UiPath.System.Activities: 22.10.x`, and (c) recommend either upgrading the package or pre-creating output variables before activity configuration + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-activity-silent-failure --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/46d9acc45e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/46d9acc45e.json deleted file mode 100644 index b2617c53b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/46d9acc45e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/703e06aea0.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/703e06aea0.json deleted file mode 100644 index cab362706a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/703e06aea0.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "UserList", - "Data": [ - { - "Id": 1042111, - "UserName": "user1", - "DisplayName": "user1", - "Type": "User", - "Roles": ["Administrator"], - "IsLicensed": true, - "LicenseType": "Administrator" - }, - { - "Id": 1042118, - "UserName": "RobotUser1", - "DisplayName": "Robot User 1", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"], - "IsLicensed": true, - "LicenseType": "Unattended", - "RobotProvision": { - "MachineName": "MOCK-HOST", - "ConnectionState": "Connected" - } - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/73b95426ba.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/73b95426ba.json deleted file mode 100644 index 6a4b60625e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/73b95426ba.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T18:02:12.299Z", - "Message": "Log Message: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at LogMessage \"Log Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T18:02:12.310Z", - "Message": "System.NullReferenceException: Object reference not set to an instance of an object.\r\n at GlobalVariablesNamespace.GlobalVariablesNamespace.Container_DesignedProcess.LambdaValue1(ActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContextUntyped(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/README.md deleted file mode 100644 index 64b464bf0e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/README.md +++ /dev/null @@ -1,86 +0,0 @@ -# Get Asset Failure — Activity Bug / Silent Failure (UiPath.System.Activities 22.10.x) - -This scenario reproduces a runtime `Get Credential` failure that is -**silent** at the activity layer — the activity itself completes -without throwing an exception, but its output variables remain `null`. -The downstream `LogMessage` activity then throws -`NullReferenceException` consuming the null outputs. The job logs do -NOT show any error from `Get Credential` itself — only the downstream -NRE. - -This is the documented `UiPath.System.Activities` **22.10.x** bug -where variables bound via `Ctrl+K` in the activity's property grid -do not receive output values at runtime. - -## What this scenario uncovers - -**Root Cause:** The project targets `UiPath.System.Activities: [22.10.5]`. -That version has a documented bug where the `Get Credential`, -`Get Asset`, and `Get Orchestrator Asset` activities silently fail to -populate output variables when those variables were created via -`Ctrl+K` during activity configuration. The fix is to upgrade the -package, or pre-create the output variables in the Variables panel -BEFORE wiring them into the activity. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-activity-bug-silent-failure.md` -(medium-confidence playbook). - -> **Why this is harder than other GetAsset scenarios:** every other -> playbook in the family is keyed off a distinctive runtime error -> from the Get Credential / Get Asset activity itself. This one is -> keyed off the **absence** of such an error combined with a -> downstream null-consumer fault AND the package version. The agent -> has to (a) trace backward from the downstream NRE to find the -> silent activity, and (b) inspect `project.json` to connect the -> pattern to the package bug. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project pinned to `UiPath.System.Activities: [22.10.5]` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. **Job logs** show `NullReferenceException` at `LogMessage "Log Message"` — NOT at `GetRobotCredential "Get Credential"`. There is no error log entry naming Get Credential. -2. **`project.json`** lists `UiPath.System.Activities: [22.10.5]` — the version with the documented Ctrl+K output-binding bug. -3. **Asset list** shows `myHiddenAsset` is present, correctly typed, with `ValueScope: "Global"` — the asset and folder are fine; the failure isn't at the asset layer. -4. **`users list`** shows `RobotUser1` has the right roles and a valid license — rules out permission / authentication / licensing issues. - -## How this differs from sibling scenarios - -| Dimension | Other GetAsset scenarios | `activity-silent-failure` (this) | -|---|---|---| -| Get Credential activity throws an error log entry? | yes | **no — completes silently** | -| Where the error log entry comes from | the Get Credential activity itself | a **downstream** activity (LogMessage) | -| Decisive evidence in `project.json`? | no | **yes — package version 22.10.x** | -| Asset layer healthy? | varies | **yes** | -| Robot / role / license healthy? | varies | **yes** | -| Matched playbook | one of get-asset-{not-found, folder-scope-mismatch, permission-denied, wrong-activity-type, per-robot-no-value, robot-not-authenticated, external-vault-failure} | `get-asset-activity-bug-silent-failure.md` | - -This is the only scenario in the family where the agent has to -troubleshoot **backward** from a non-asset-shaped error (NRE at -LogMessage) AND inspect `project.json` to find a package-version -anchor. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) recognize that Get Credential completed silently, (b) name the package version `UiPath.System.Activities: 22.10.x`, and (c) recommend either upgrading the package or pre-creating output variables before activity configuration - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-activity-silent-failure --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/a17b40918f.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/a17b40918f.json deleted file mode 100644 index 9f85a10e1b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/a17b40918f.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-13T18:02:11.804Z", - "Message": "SettingsAssetFetcher execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-13T18:02:12.118Z", - "Message": "Get Credential: execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-13T18:02:12.221Z", - "Message": "Get Credential: execution ended (no exception)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-13T18:02:12.244Z", - "Message": "Log Message: execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T18:02:12.299Z", - "Message": "Log Message: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at LogMessage \"Log Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T18:02:12.310Z", - "Message": "System.NullReferenceException: Object reference not set to an instance of an object.\r\n at GlobalVariablesNamespace.GlobalVariablesNamespace.Container_DesignedProcess.LambdaValue1(ActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContextUntyped(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T18:02:12.341Z", - "Message": "SettingsAssetFetcher execution ended" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/c71e761bef.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/c71e761bef.json deleted file mode 100644 index 83667e34ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/c71e761bef.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d", - "State": "Faulted", - "ProcessName": "SettingsAssetFetcher", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T18:02:11.040Z", - "EndTime": "2026-05-13T18:02:12.318Z", - "CreationTime": "2026-05-13T18:02:10.815Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/e31450ec93.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/e31450ec93.json deleted file mode 100644 index bdc48a887a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/e31450ec93.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/f9d1c2249e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/f9d1c2249e.json deleted file mode 100644 index 0c9d52d4a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/f9d1c2249e.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d", - "StartTime": "2026-05-13T18:02:11.040Z", - "EndTime": "2026-05-13T18:02:12.318Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "2b4d6f8a-3c1e-4f5b-9d7c-8a1b2c3d4e5f", - "Info": "Log Message: Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at LogMessage \"Log Message\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-13T18:02:10.815Z", - "ReleaseName": "SettingsAssetFetcher", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5039207, - "OrganizationUnitId": 7791625, - "FolderKey": "5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T18:02:12.318Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d", - "Id": 670980413 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/manifest.json deleted file mode 100644 index 2b87e6f7ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/m/r/manifest.json +++ /dev/null @@ -1,88 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-activity-silent-failure scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e31450ec93.json" - }, - { - "match": "or folders list --output json", - "file": "e31450ec93.json" - }, - { - "match": "or jobs list --folder-key \"5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a\" --state Faulted --output json", - "file": "c71e761bef.json" - }, - { - "match": "or jobs list --folder-key 5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a --state Faulted --output json", - "file": "c71e761bef.json" - }, - { - "match": "or jobs list --process-name SettingsAssetFetcher", - "file": "c71e761bef.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "c71e761bef.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "c71e761bef.json" - }, - { - "match": "or jobs get \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\" --output json", - "file": "f9d1c2249e.json" - }, - { - "match": "or jobs get 7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d --output json", - "file": "f9d1c2249e.json" - }, - { - "match": "or jobs logs \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\" --level Error --output json", - "file": "73b95426ba.json" - }, - { - "match": "or jobs logs 7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d --level Error --output json", - "file": "73b95426ba.json" - }, - { - "match": "or jobs logs \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\" --output json", - "file": "a17b40918f.json" - }, - { - "match": "or jobs logs 7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d --output json", - "file": "a17b40918f.json" - }, - { - "match": "or assets list --folder-key 5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-key \"5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a\" --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "46d9acc45e.json" - }, - { - "match": "or users list --output json", - "file": "703e06aea0.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/uip-fixture.json new file mode 100644 index 0000000000..24766bfe05 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/data/uip-fixture.json @@ -0,0 +1,166 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SettingsAssetFetcher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T18:02:11.040Z\",\r\n \"EndTime\": \"2026-05-13T18:02:12.318Z\",\r\n \"CreationTime\": \"2026-05-13T18:02:10.815Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SettingsAssetFetcher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T18:02:11.040Z\",\r\n \"EndTime\": \"2026-05-13T18:02:12.318Z\",\r\n \"CreationTime\": \"2026-05-13T18:02:10.815Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.299Z\",\r\n \"Message\": \"Log Message: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at LogMessage \\\"Log Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.310Z\",\r\n \"Message\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at GlobalVariablesNamespace.GlobalVariablesNamespace.Container_DesignedProcess.LambdaValue1(ActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContextUntyped(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "SettingsAssetFetcher" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SettingsAssetFetcher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T18:02:11.040Z\",\r\n \"EndTime\": \"2026-05-13T18:02:12.318Z\",\r\n \"CreationTime\": \"2026-05-13T18:02:10.815Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"SettingsAssetFetcher\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T18:02:11.040Z\",\r\n \"EndTime\": \"2026-05-13T18:02:12.318Z\",\r\n \"CreationTime\": \"2026-05-13T18:02:10.815Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\",\r\n \"StartTime\": \"2026-05-13T18:02:11.040Z\",\r\n \"EndTime\": \"2026-05-13T18:02:12.318Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"2b4d6f8a-3c1e-4f5b-9d7c-8a1b2c3d4e5f\",\r\n \"Info\": \"Log Message: Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at LogMessage \\\"Log Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-13T18:02:10.815Z\",\r\n \"ReleaseName\": \"SettingsAssetFetcher\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5039207,\r\n \"OrganizationUnitId\": 7791625,\r\n \"FolderKey\": \"5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T18:02:12.318Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d\",\r\n \"Id\": 670980413\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7d2e5f3c-1a6b-4e9f-8c4d-3e2f1a0b9c8d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T18:02:11.804Z\",\r\n \"Message\": \"SettingsAssetFetcher execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.118Z\",\r\n \"Message\": \"Get Credential: execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.221Z\",\r\n \"Message\": \"Get Credential: execution ended (no exception)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.244Z\",\r\n \"Message\": \"Log Message: execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.299Z\",\r\n \"Message\": \"Log Message: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at LogMessage \\\"Log Message\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.310Z\",\r\n \"Message\": \"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at GlobalVariablesNamespace.GlobalVariablesNamespace.Container_DesignedProcess.LambdaValue1(ActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContextUntyped(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T18:02:12.341Z\",\r\n \"Message\": \"SettingsAssetFetcher execution ended\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a8c4d3e-9f2b-4a6c-8e1f-2b3c4d5e6f7a\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserList\",\r\n \"Data\": [\r\n {\r\n \"Id\": 1042111,\r\n \"UserName\": \"user1\",\r\n \"DisplayName\": \"user1\",\r\n \"Type\": \"User\",\r\n \"Roles\": [\"Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Administrator\"\r\n },\r\n {\r\n \"Id\": 1042118,\r\n \"UserName\": \"RobotUser1\",\r\n \"DisplayName\": \"Robot User 1\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Unattended\",\r\n \"RobotProvision\": {\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"ConnectionState\": \"Connected\"\r\n }\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/task.yaml index 0866d1b79c..d8223f21f2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-activity-silent-failure/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/README.md new file mode 100644 index 0000000000..82b1a6b60e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/README.md @@ -0,0 +1,82 @@ +# Get Asset Failure — External Credential Store Failure + +This scenario reproduces a runtime `Get Credential` failure caused by +an **external credential store** (CyberArk, Azure Key Vault, Thycotic, +or other vault) being unreachable or misconfigured. The asset itself +is correctly defined in Orchestrator — its value just lives in a +remote vault that Orchestrator can't reach. Orchestrator returns +**error code 2304** / "Failed to read from Credential Store type +'CyberArk'". + +## What this scenario uncovers + +**Root Cause:** The `Get Credential` activity targets a Credential +asset (`MyCyberArkSecret`) whose value is stored in an external +CyberArk vault. The asset's `CredentialStoreId` points at a non-default +store named "Production CyberArk Store" of type `CyberArk`. The vault +endpoint is currently unreachable from Orchestrator (network issue, +endpoint moved, or configuration drift on the Orchestrator side). +Orchestrator therefore rejects the read with error code 2304. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-external-vault-failure.md` +(medium-confidence playbook). + +> **Why "medium-confidence":** error codes 2303 / 2304 are unique to +> external-vault failures but the underlying cause has many +> sub-branches per the playbook — network unreachable, CyberArk FIPS +> mode mismatch, Azure Key Vault IP not whitelisted, Thycotic +> integration URL wrong, plugin configuration incomplete. The agent +> must identify the vault type from the error message and recommend +> the right diagnosis path. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Get Credential activity correctly configured at every workflow layer | + +The decisive evidence: + +1. The error log: `Failed to read from Credential Store type 'CyberArk'. Error code: 2304`. +2. The asset list shows `MyCyberArkSecret` with a non-null `CredentialStoreId` and `ExternalName` — proof the asset's value is stored externally. +3. The credential-stores list (`uip orch credential-stores list`) returns "Production CyberArk Store" with `Type: "CyberArk"` matching the asset's binding. + +## How this differs from sibling scenarios + +| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` | `per-robot-no-value` | `robot-not-authenticated` | `external-vault-failure` (this) | +|---|---|---|---|---|---|---|---| +| Asset present? | no (typo) | n/a | yes | yes (wrong type) | yes | yes | yes | +| Folder present? | yes | no | yes | yes | yes | yes | yes | +| Permissions correct? | n/a | n/a | no | n/a | n/a | n/a | yes | +| Robot licensed? | n/a | n/a | n/a | n/a | n/a | no | yes | +| External vault backing? | n/a | n/a | n/a | n/a | n/a | n/a | **yes — CyberArk (unreachable)** | +| Error code anchor | 1002 | 1100 | 0 (HTTP 403) | "Invalid asset type" | n/a | 0 (HTTP 401) | **2304** | +| Matched playbook | not-found | folder-scope-mismatch | permission-denied | wrong-activity-type | per-robot-no-value | robot-not-authenticated | external-vault-failure | + +This scenario is the first where the failing layer is **outside +Orchestrator entirely** — it's the external vault product. The agent +must recommend connectivity / configuration checks on the vault side, +not changes to the asset, the folder, the robot, or the workflow. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) name the vault type (`CyberArk`), (b) recognize the asset is backed by an external store, and (c) recommend checking vault connectivity / configuration on the Orchestrator-to-vault path + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-external-vault-failure --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/2804c3e55b.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/2804c3e55b.json deleted file mode 100644 index a2aca5da7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/2804c3e55b.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "CredentialStoreList", - "Data": [ - { - "Id": "00000000-0000-0000-0000-000000000001", - "Name": "Orchestrator Database", - "Type": "Database", - "IsDefault": true, - "Description": "Default Orchestrator-managed credential store." - }, - { - "Id": "a3e7b2c5-4d8f-49a1-b6c2-8e7d1f3a5b9c", - "Name": "Production CyberArk Store", - "Type": "CyberArk", - "IsDefault": false, - "Description": "External CyberArk vault hosting production service-account credentials.", - "Configuration": { - "AppId": "UiPath-Orchestrator-Prod", - "Safe": "Production", - "Folder": "Root", - "WebServiceUrl": "https://cyberark.prod.internal/AIMWebService" - } - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/6134aabc50.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/6134aabc50.json deleted file mode 100644 index bb0ca35919..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/6134aabc50.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T15:42:09.380Z", - "Message": "[Get Credential] Status code: 500 (Internal Server Error). Orchestrator response: \r\nFailed to read from Credential Store type 'CyberArk'. Error code: 2304" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T15:42:09.395Z", - "Message": "Failed to read from Credential Store type 'CyberArk'. Error code: 2304\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/703e06aea0.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/703e06aea0.json deleted file mode 100644 index cab362706a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/703e06aea0.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "UserList", - "Data": [ - { - "Id": 1042111, - "UserName": "user1", - "DisplayName": "user1", - "Type": "User", - "Roles": ["Administrator"], - "IsLicensed": true, - "LicenseType": "Administrator" - }, - { - "Id": 1042118, - "UserName": "RobotUser1", - "DisplayName": "Robot User 1", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"], - "IsLicensed": true, - "LicenseType": "Unattended", - "RobotProvision": { - "MachineName": "MOCK-HOST", - "ConnectionState": "Connected" - } - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/8794ad9f3c.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/8794ad9f3c.json deleted file mode 100644 index 8e0b649ae7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/8794ad9f3c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c", - "State": "Faulted", - "ProcessName": "VaultCredentialReader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T15:42:08.041Z", - "EndTime": "2026-05-13T15:42:09.402Z", - "CreationTime": "2026-05-13T15:42:07.812Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/918ca7cb85.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/918ca7cb85.json deleted file mode 100644 index 7f185d9585..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/918ca7cb85.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "e2f3a4b5-c6d7-8901-defa-2345678901cd", - "Name": "MyCyberArkSecret", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset backed by the Production CyberArk Store. Value is held in CyberArk, not in Orchestrator's database.", - "CredentialStoreId": "a3e7b2c5-4d8f-49a1-b6c2-8e7d1f3a5b9c", - "ExternalName": "ProductionDB-ServiceAccount" - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures.", - "CredentialStoreId": null, - "ExternalName": null - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/README.md deleted file mode 100644 index b2eec433aa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/README.md +++ /dev/null @@ -1,81 +0,0 @@ -# Get Asset Failure — External Credential Store Failure - -This scenario reproduces a runtime `Get Credential` failure caused by -an **external credential store** (CyberArk, Azure Key Vault, Thycotic, -or other vault) being unreachable or misconfigured. The asset itself -is correctly defined in Orchestrator — its value just lives in a -remote vault that Orchestrator can't reach. Orchestrator returns -**error code 2304** / "Failed to read from Credential Store type -'CyberArk'". - -## What this scenario uncovers - -**Root Cause:** The `Get Credential` activity targets a Credential -asset (`MyCyberArkSecret`) whose value is stored in an external -CyberArk vault. The asset's `CredentialStoreId` points at a non-default -store named "Production CyberArk Store" of type `CyberArk`. The vault -endpoint is currently unreachable from Orchestrator (network issue, -endpoint moved, or configuration drift on the Orchestrator side). -Orchestrator therefore rejects the read with error code 2304. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-external-vault-failure.md` -(medium-confidence playbook). - -> **Why "medium-confidence":** error codes 2303 / 2304 are unique to -> external-vault failures but the underlying cause has many -> sub-branches per the playbook — network unreachable, CyberArk FIPS -> mode mismatch, Azure Key Vault IP not whitelisted, Thycotic -> integration URL wrong, plugin configuration incomplete. The agent -> must identify the vault type from the error message and recommend -> the right diagnosis path. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Get Credential activity correctly configured at every workflow layer | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. The error log: `Failed to read from Credential Store type 'CyberArk'. Error code: 2304`. -2. The asset list shows `MyCyberArkSecret` with a non-null `CredentialStoreId` and `ExternalName` — proof the asset's value is stored externally. -3. The credential-stores list (`uip orch credential-stores list`) returns "Production CyberArk Store" with `Type: "CyberArk"` matching the asset's binding. - -## How this differs from sibling scenarios - -| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` | `per-robot-no-value` | `robot-not-authenticated` | `external-vault-failure` (this) | -|---|---|---|---|---|---|---|---| -| Asset present? | no (typo) | n/a | yes | yes (wrong type) | yes | yes | yes | -| Folder present? | yes | no | yes | yes | yes | yes | yes | -| Permissions correct? | n/a | n/a | no | n/a | n/a | n/a | yes | -| Robot licensed? | n/a | n/a | n/a | n/a | n/a | no | yes | -| External vault backing? | n/a | n/a | n/a | n/a | n/a | n/a | **yes — CyberArk (unreachable)** | -| Error code anchor | 1002 | 1100 | 0 (HTTP 403) | "Invalid asset type" | n/a | 0 (HTTP 401) | **2304** | -| Matched playbook | not-found | folder-scope-mismatch | permission-denied | wrong-activity-type | per-robot-no-value | robot-not-authenticated | external-vault-failure | - -This scenario is the first where the failing layer is **outside -Orchestrator entirely** — it's the external vault product. The agent -must recommend connectivity / configuration checks on the vault side, -not changes to the asset, the folder, the robot, or the workflow. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) name the vault type (`CyberArk`), (b) recognize the asset is backed by an external store, and (c) recommend checking vault connectivity / configuration on the Orchestrator-to-vault path - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-external-vault-failure --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/e210f4c018.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/e210f4c018.json deleted file mode 100644 index a2d173a860..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/e210f4c018.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c", - "StartTime": "2026-05-13T15:42:08.041Z", - "EndTime": "2026-05-13T15:42:09.402Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9f2b3e4c-5a1d-4e6f-9b8c-7d2a3e4f5b6c", - "Info": "Get Credential: Status code: 500 (Internal Server Error). Orchestrator response: Failed to read from Credential Store type 'CyberArk'. Error code: 2304\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.Activities.OrchestratorCommunicationException: Failed to read from Credential Store type 'CyberArk'. Error code: 2304", - "CreationTime": "2026-05-13T15:42:07.812Z", - "ReleaseName": "VaultCredentialReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5035218, - "OrganizationUnitId": 7791624, - "FolderKey": "4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T15:42:09.402Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c", - "Id": 670974502 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/e7a72c8342.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/e7a72c8342.json deleted file mode 100644 index 23d6061535..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/e7a72c8342.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/manifest.json deleted file mode 100644 index f0df3ceb77..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/m/r/manifest.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-external-vault-failure scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e7a72c8342.json" - }, - { - "match": "or folders list --output json", - "file": "e7a72c8342.json" - }, - { - "match": "or jobs list --folder-key \"4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a\" --state Faulted --output json", - "file": "8794ad9f3c.json" - }, - { - "match": "or jobs list --folder-key 4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a --state Faulted --output json", - "file": "8794ad9f3c.json" - }, - { - "match": "or jobs list --process-name VaultCredentialReader", - "file": "8794ad9f3c.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "8794ad9f3c.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "8794ad9f3c.json" - }, - { - "match": "or jobs get \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\" --output json", - "file": "e210f4c018.json" - }, - { - "match": "or jobs get 6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c --output json", - "file": "e210f4c018.json" - }, - { - "match": "or jobs logs \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\" --level Error --output json", - "file": "6134aabc50.json" - }, - { - "match": "or jobs logs 6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c --level Error --output json", - "file": "6134aabc50.json" - }, - { - "match": "or assets list --folder-key 4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a --output json", - "file": "918ca7cb85.json" - }, - { - "match": "or assets list --folder-key \"4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a\" --output json", - "file": "918ca7cb85.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "918ca7cb85.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "918ca7cb85.json" - }, - { - "match": "or users list --output json", - "file": "703e06aea0.json" - }, - { - "match": "or credential-stores list --output json", - "file": "2804c3e55b.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/uip-fixture.json new file mode 100644 index 0000000000..09049ab13f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/data/uip-fixture.json @@ -0,0 +1,165 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VaultCredentialReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T15:42:08.041Z\",\r\n \"EndTime\": \"2026-05-13T15:42:09.402Z\",\r\n \"CreationTime\": \"2026-05-13T15:42:07.812Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VaultCredentialReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T15:42:08.041Z\",\r\n \"EndTime\": \"2026-05-13T15:42:09.402Z\",\r\n \"CreationTime\": \"2026-05-13T15:42:07.812Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T15:42:09.380Z\",\r\n \"Message\": \"[Get Credential] Status code: 500 (Internal Server Error). Orchestrator response: \\r\\nFailed to read from Credential Store type 'CyberArk'. Error code: 2304\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T15:42:09.395Z\",\r\n \"Message\": \"Failed to read from Credential Store type 'CyberArk'. Error code: 2304\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e2f3a4b5-c6d7-8901-defa-2345678901cd\",\r\n \"Name\": \"MyCyberArkSecret\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset backed by the Production CyberArk Store. Value is held in CyberArk, not in Orchestrator's database.\",\r\n \"CredentialStoreId\": \"a3e7b2c5-4d8f-49a1-b6c2-8e7d1f3a5b9c\",\r\n \"ExternalName\": \"ProductionDB-ServiceAccount\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\",\r\n \"CredentialStoreId\": null,\r\n \"ExternalName\": null\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "VaultCredentialReader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VaultCredentialReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T15:42:08.041Z\",\r\n \"EndTime\": \"2026-05-13T15:42:09.402Z\",\r\n \"CreationTime\": \"2026-05-13T15:42:07.812Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"VaultCredentialReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T15:42:08.041Z\",\r\n \"EndTime\": \"2026-05-13T15:42:09.402Z\",\r\n \"CreationTime\": \"2026-05-13T15:42:07.812Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e2f3a4b5-c6d7-8901-defa-2345678901cd\",\r\n \"Name\": \"MyCyberArkSecret\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset backed by the Production CyberArk Store. Value is held in CyberArk, not in Orchestrator's database.\",\r\n \"CredentialStoreId\": \"a3e7b2c5-4d8f-49a1-b6c2-8e7d1f3a5b9c\",\r\n \"ExternalName\": \"ProductionDB-ServiceAccount\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\",\r\n \"CredentialStoreId\": null,\r\n \"ExternalName\": null\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e2f3a4b5-c6d7-8901-defa-2345678901cd\",\r\n \"Name\": \"MyCyberArkSecret\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset backed by the Production CyberArk Store. Value is held in CyberArk, not in Orchestrator's database.\",\r\n \"CredentialStoreId\": \"a3e7b2c5-4d8f-49a1-b6c2-8e7d1f3a5b9c\",\r\n \"ExternalName\": \"ProductionDB-ServiceAccount\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\",\r\n \"CredentialStoreId\": null,\r\n \"ExternalName\": null\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"StartTime\": \"2026-05-13T15:42:08.041Z\",\r\n \"EndTime\": \"2026-05-13T15:42:09.402Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9f2b3e4c-5a1d-4e6f-9b8c-7d2a3e4f5b6c\",\r\n \"Info\": \"Get Credential: Status code: 500 (Internal Server Error). Orchestrator response: Failed to read from Credential Store type 'CyberArk'. Error code: 2304\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorCommunicationException: Failed to read from Credential Store type 'CyberArk'. Error code: 2304\",\r\n \"CreationTime\": \"2026-05-13T15:42:07.812Z\",\r\n \"ReleaseName\": \"VaultCredentialReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5035218,\r\n \"OrganizationUnitId\": 7791624,\r\n \"FolderKey\": \"4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T15:42:09.402Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"6c9e3a4b-1f5d-4e8c-9b6a-2d3e4f5a6b7c\",\r\n \"Id\": 670974502\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4f7a9b3d-2e8c-4f6b-9d5a-1c2b3d4e5f6a\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserList\",\r\n \"Data\": [\r\n {\r\n \"Id\": 1042111,\r\n \"UserName\": \"user1\",\r\n \"DisplayName\": \"user1\",\r\n \"Type\": \"User\",\r\n \"Roles\": [\"Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Administrator\"\r\n },\r\n {\r\n \"Id\": 1042118,\r\n \"UserName\": \"RobotUser1\",\r\n \"DisplayName\": \"Robot User 1\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Unattended\",\r\n \"RobotProvision\": {\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"ConnectionState\": \"Connected\"\r\n }\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "credential-stores", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"CredentialStoreList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"00000000-0000-0000-0000-000000000001\",\r\n \"Name\": \"Orchestrator Database\",\r\n \"Type\": \"Database\",\r\n \"IsDefault\": true,\r\n \"Description\": \"Default Orchestrator-managed credential store.\"\r\n },\r\n {\r\n \"Id\": \"a3e7b2c5-4d8f-49a1-b6c2-8e7d1f3a5b9c\",\r\n \"Name\": \"Production CyberArk Store\",\r\n \"Type\": \"CyberArk\",\r\n \"IsDefault\": false,\r\n \"Description\": \"External CyberArk vault hosting production service-account credentials.\",\r\n \"Configuration\": {\r\n \"AppId\": \"UiPath-Orchestrator-Prod\",\r\n \"Safe\": \"Production\",\r\n \"Folder\": \"Root\",\r\n \"WebServiceUrl\": \"https://cyberark.prod.internal/AIMWebService\"\r\n }\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/task.yaml index 91411cffec..4b205e2dcc 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-external-vault-failure/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/README.md new file mode 100644 index 0000000000..e6d9b135a9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/README.md @@ -0,0 +1,65 @@ +# Get Asset Failure — Folder Scope Mismatch + +This scenario reproduces a runtime `Get Credential` failure caused by a +`FolderPath` property pointing to a folder that does not exist in the +tenant. Orchestrator returns error code **1100** ("Folder does not +exist or the user does not have access to the folder"). + +## What this scenario uncovers + +**Root Cause:** The `Get Credential` activity in `Main.xaml` has +`FolderPath="OldDevFolder"`. The tenant folders list returns only +`Remote Debugging` (where the job actually runs) and `Shared` — there +is no folder named `OldDevFolder`. Orchestrator cannot resolve the +asset because the targeted folder does not exist. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-folder-scope-mismatch.md` +(the "OrchestratorFolderPath set to a wrong or nonexistent folder" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — derived from the `getasset-name-mismatch` scenario with `AssetName` fixed (correct spelling) and `FolderPath` mutated to a non-existent folder | + +> **Note on fixtures.** Like `getasset-name-mismatch`, fixtures here +> were authored from the documented playbook signature rather than +> captured from a real `.local/investigations/` session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression signal. + +## Why this is different from `getasset-name-mismatch` + +| Dimension | `getasset-name-mismatch` | `getasset-folder-scope-mismatch` | +|---|---|---| +| AssetName property | typo (`myHiddenAset`) | correctly spelled (`myHiddenAsset`) | +| FolderPath property | matches a real folder (`Remote Debugging`) | points to a non-existent folder (`OldDevFolder`) | +| Orchestrator error code | **1002** (asset not found) | **1100** (folder doesn't exist / no access) | +| Playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | + +Both scenarios use the same `Get Credential` activity shape, so they +exercise the troubleshooting agent's ability to discriminate between +adjacent failure modes — name-level vs folder-level lookup failure. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Specifically, the agent must name `OldDevFolder` and confirm it does not exist in the tenant's folder list + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-folder-scope-mismatch --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/243a6d8ad0.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/243a6d8ad0.json deleted file mode 100644 index 855f1944c3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/243a6d8ad0.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T09:18:44.355Z", - "Message": "[Get Credential] Status code: 403 (Forbidden). Orchestrator response: \r\nFolder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T09:18:44.371Z", - "Message": "Folder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/46d9acc45e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/46d9acc45e.json deleted file mode 100644 index b2617c53b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/46d9acc45e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/7639697a15.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/7639697a15.json deleted file mode 100644 index 8a4971c387..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/7639697a15.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4", - "StartTime": "2026-05-12T09:18:42.110Z", - "EndTime": "2026-05-12T09:18:44.380Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5a1d3f7e-2c8b-4a9f-9e3d-7b8c0a1d2e3f", - "Info": "Get Credential: Status code: 403 (Forbidden). Orchestrator response: Folder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ServiceModel.FaultException: Folder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100", - "CreationTime": "2026-05-12T09:18:41.870Z", - "ReleaseName": "ConfigAssetReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5018612, - "OrganizationUnitId": 7791619, - "FolderKey": "6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T09:18:44.380Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4", - "Id": 670934211 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/README.md deleted file mode 100644 index cd9176476d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/README.md +++ /dev/null @@ -1,64 +0,0 @@ -# Get Asset Failure — Folder Scope Mismatch - -This scenario reproduces a runtime `Get Credential` failure caused by a -`FolderPath` property pointing to a folder that does not exist in the -tenant. Orchestrator returns error code **1100** ("Folder does not -exist or the user does not have access to the folder"). - -## What this scenario uncovers - -**Root Cause:** The `Get Credential` activity in `Main.xaml` has -`FolderPath="OldDevFolder"`. The tenant folders list returns only -`Remote Debugging` (where the job actually runs) and `Shared` — there -is no folder named `OldDevFolder`. Orchestrator cannot resolve the -asset because the targeted folder does not exist. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-folder-scope-mismatch.md` -(the "OrchestratorFolderPath set to a wrong or nonexistent folder" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | synthesized UiPath project — derived from the `getasset-name-mismatch` scenario with `AssetName` fixed (correct spelling) and `FolderPath` mutated to a non-existent folder | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like `getasset-name-mismatch`, fixtures here -> were authored from the documented playbook signature rather than -> captured from a real `.local/investigations/` session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression signal. - -## Why this is different from `getasset-name-mismatch` - -| Dimension | `getasset-name-mismatch` | `getasset-folder-scope-mismatch` | -|---|---|---| -| AssetName property | typo (`myHiddenAset`) | correctly spelled (`myHiddenAsset`) | -| FolderPath property | matches a real folder (`Remote Debugging`) | points to a non-existent folder (`OldDevFolder`) | -| Orchestrator error code | **1002** (asset not found) | **1100** (folder doesn't exist / no access) | -| Playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | - -Both scenarios use the same `Get Credential` activity shape, so they -exercise the troubleshooting agent's ability to discriminate between -adjacent failure modes — name-level vs folder-level lookup failure. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Specifically, the agent must name `OldDevFolder` and confirm it does not exist in the tenant's folder list - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-folder-scope-mismatch --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/aed1d35dca.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/aed1d35dca.json deleted file mode 100644 index 8485af5325..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/aed1d35dca.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4", - "State": "Faulted", - "ProcessName": "ConfigAssetReader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T09:18:42.110Z", - "EndTime": "2026-05-12T09:18:44.380Z", - "CreationTime": "2026-05-12T09:18:41.870Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/b4481765d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/b4481765d3.json deleted file mode 100644 index 20425b9e95..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/b4481765d3.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/manifest.json deleted file mode 100644 index b055d78617..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-folder-scope-mismatch scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "b4481765d3.json" - }, - { - "match": "or folders list --output json", - "file": "b4481765d3.json" - }, - { - "match": "or jobs list --folder-key \"6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f\" --state Faulted --output json", - "file": "aed1d35dca.json" - }, - { - "match": "or jobs list --folder-key 6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f --state Faulted --output json", - "file": "aed1d35dca.json" - }, - { - "match": "or jobs list --process-name ConfigAssetReader", - "file": "aed1d35dca.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "aed1d35dca.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "aed1d35dca.json" - }, - { - "match": "or jobs get \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\" --output json", - "file": "7639697a15.json" - }, - { - "match": "or jobs get 8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4 --output json", - "file": "7639697a15.json" - }, - { - "match": "or jobs logs \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\" --level Error --output json", - "file": "243a6d8ad0.json" - }, - { - "match": "or jobs logs 8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4 --level Error --output json", - "file": "243a6d8ad0.json" - }, - { - "match": "or assets list --folder-key 6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-key \"6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f\" --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "46d9acc45e.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..6638be90b4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/data/uip-fixture.json @@ -0,0 +1,141 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ConfigAssetReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T09:18:42.110Z\",\r\n \"EndTime\": \"2026-05-12T09:18:44.380Z\",\r\n \"CreationTime\": \"2026-05-12T09:18:41.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ConfigAssetReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T09:18:42.110Z\",\r\n \"EndTime\": \"2026-05-12T09:18:44.380Z\",\r\n \"CreationTime\": \"2026-05-12T09:18:41.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T09:18:44.355Z\",\r\n \"Message\": \"[Get Credential] Status code: 403 (Forbidden). Orchestrator response: \\r\\nFolder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T09:18:44.371Z\",\r\n \"Message\": \"Folder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "ConfigAssetReader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ConfigAssetReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T09:18:42.110Z\",\r\n \"EndTime\": \"2026-05-12T09:18:44.380Z\",\r\n \"CreationTime\": \"2026-05-12T09:18:41.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ConfigAssetReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T09:18:42.110Z\",\r\n \"EndTime\": \"2026-05-12T09:18:44.380Z\",\r\n \"CreationTime\": \"2026-05-12T09:18:41.870Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\",\r\n \"StartTime\": \"2026-05-12T09:18:42.110Z\",\r\n \"EndTime\": \"2026-05-12T09:18:44.380Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5a1d3f7e-2c8b-4a9f-9e3d-7b8c0a1d2e3f\",\r\n \"Info\": \"Get Credential: Status code: 403 (Forbidden). Orchestrator response: Folder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ServiceModel.FaultException: Folder 'OldDevFolder' does not exist or the user does not have access to the folder. Error code: 1100\",\r\n \"CreationTime\": \"2026-05-12T09:18:41.870Z\",\r\n \"ReleaseName\": \"ConfigAssetReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5018612,\r\n \"OrganizationUnitId\": 7791619,\r\n \"FolderKey\": \"6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-12T09:18:44.380Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8e4f3b2c-1a9d-5e6f-b7c8-d9e0f1a2b3c4\",\r\n \"Id\": 670934211\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6d8c5e3f-2a1b-4c9d-8e7f-0a1b2c3d4e5f\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/task.yaml index 17dba71b80..13b37be454 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-folder-scope-mismatch/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/README.md new file mode 100644 index 0000000000..2e60e4a1ae --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/README.md @@ -0,0 +1,54 @@ +# Get Asset Failure — Name Mismatch (Typo) + +This scenario reproduces a runtime `Get Credential` failure caused by an +**AssetName typo** in the workflow XAML. Orchestrator returns HTTP 404 / +error code 1002 because the asset literally named in the activity does +not exist — but a near-miss asset with the corrected spelling does +exist in the same folder. + +## What this scenario uncovers + +**Root Cause:** The `Get Credential` activity in `Main.xaml` references +`AssetName="myHiddenAset"` (missing an "s"). The intended asset in the +`Remote Debugging` folder is `myHiddenAsset`. Orchestrator's exact-match +asset lookup returns 404. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-not-found.md` +(specifically the "case or spelling difference" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json` from `C:\Temp\GetAsset Scenarios\name mismatch`) | + +> **Note on fixtures.** Unlike `rpa-preflight-failure` and +> `faulted_excel_o365`, the fixtures here were authored from the +> documented playbook signature rather than captured from a real +> `.local/investigations/` session. Before treating this test's score +> as a regression signal, replay against a real failed job and +> regenerate via `_shared/scripts/generate_scenario.py` — see +> `tests/tasks/uipath-troubleshoot/CLAUDE.md`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Specifically, the agent must name the typo (`myHiddenAset` vs `myHiddenAsset`), not just "asset missing" + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-name-mismatch --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/0e83076f2e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/0e83076f2e.json deleted file mode 100644 index f8fe81bc6b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/0e83076f2e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/46d9acc45e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/46d9acc45e.json deleted file mode 100644 index b2617c53b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/46d9acc45e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/7b26c0dde0.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/7b26c0dde0.json deleted file mode 100644 index b8c65d760c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/7b26c0dde0.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3", - "StartTime": "2026-05-10T14:32:11.080Z", - "EndTime": "2026-05-10T14:32:13.420Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "8c2f4e6a-1b3d-4f5a-9c2e-7d8f0a1b3c4d", - "Info": "Get Credential: Status code: 404 (Not Found). Orchestrator response: Could not find the asset 'myHiddenAset'. Error code: 1002\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ServiceModel.FaultException: Could not find the asset 'myHiddenAset'. Error code: 1002", - "CreationTime": "2026-05-10T14:32:10.825Z", - "ReleaseName": "AssetLookupRunner", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5012840, - "OrganizationUnitId": 7791618, - "FolderKey": "5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-10T14:32:13.420Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3", - "Id": 670920718 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/7edc8b39fd.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/7edc8b39fd.json deleted file mode 100644 index ea04a47c0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/7edc8b39fd.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-10T14:32:13.371Z", - "Message": "[Get Credential] Status code: 404 (Not Found). Orchestrator response: \r\nCould not find the asset 'myHiddenAset'. Error code: 1002" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-10T14:32:13.412Z", - "Message": "Could not find the asset 'myHiddenAset'. Error code: 1002\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/README.md deleted file mode 100644 index df9aa98b01..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Get Asset Failure — Name Mismatch (Typo) - -This scenario reproduces a runtime `Get Credential` failure caused by an -**AssetName typo** in the workflow XAML. Orchestrator returns HTTP 404 / -error code 1002 because the asset literally named in the activity does -not exist — but a near-miss asset with the corrected spelling does -exist in the same folder. - -## What this scenario uncovers - -**Root Cause:** The `Get Credential` activity in `Main.xaml` references -`AssetName="myHiddenAset"` (missing an "s"). The intended asset in the -`Remote Debugging` folder is `myHiddenAsset`. Orchestrator's exact-match -asset lookup returns 404. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-not-found.md` -(specifically the "case or spelling difference" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | snapshot of the failing UiPath project (`Main.xaml` + `project.json` from `C:\Temp\GetAsset Scenarios\name mismatch`) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Unlike `rpa-preflight-failure` and -> `faulted_excel_o365`, the fixtures here were authored from the -> documented playbook signature rather than captured from a real -> `.local/investigations/` session. Before treating this test's score -> as a regression signal, replay against a real failed job and -> regenerate via `_shared/scripts/generate_scenario.py` — see -> `tests/tasks/uipath-troubleshoot/CLAUDE.md`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Specifically, the agent must name the typo (`myHiddenAset` vs `myHiddenAsset`), not just "asset missing" - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-name-mismatch --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/a3f5a7328b.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/a3f5a7328b.json deleted file mode 100644 index b79c4d8ee3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/a3f5a7328b.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3", - "State": "Faulted", - "ProcessName": "AssetLookupRunner", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-10T14:32:11.080Z", - "EndTime": "2026-05-10T14:32:13.420Z", - "CreationTime": "2026-05-10T14:32:10.825Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/manifest.json deleted file mode 100644 index 8153c37507..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-name-mismatch scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "0e83076f2e.json" - }, - { - "match": "or folders list --output json", - "file": "0e83076f2e.json" - }, - { - "match": "or jobs list --folder-key \"5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d\" --state Faulted --output json", - "file": "a3f5a7328b.json" - }, - { - "match": "or jobs list --folder-key 5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d --state Faulted --output json", - "file": "a3f5a7328b.json" - }, - { - "match": "or jobs list --process-name AssetLookupRunner", - "file": "a3f5a7328b.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "a3f5a7328b.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "a3f5a7328b.json" - }, - { - "match": "or jobs get \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\" --output json", - "file": "7b26c0dde0.json" - }, - { - "match": "or jobs get 7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3 --output json", - "file": "7b26c0dde0.json" - }, - { - "match": "or jobs logs \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\" --level Error --output json", - "file": "7edc8b39fd.json" - }, - { - "match": "or jobs logs 7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3 --level Error --output json", - "file": "7edc8b39fd.json" - }, - { - "match": "or assets list --folder-key 5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-key \"5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d\" --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "46d9acc45e.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..874d044a08 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/data/uip-fixture.json @@ -0,0 +1,141 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetLookupRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-10T14:32:11.080Z\",\r\n \"EndTime\": \"2026-05-10T14:32:13.420Z\",\r\n \"CreationTime\": \"2026-05-10T14:32:10.825Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetLookupRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-10T14:32:11.080Z\",\r\n \"EndTime\": \"2026-05-10T14:32:13.420Z\",\r\n \"CreationTime\": \"2026-05-10T14:32:10.825Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-10T14:32:13.371Z\",\r\n \"Message\": \"[Get Credential] Status code: 404 (Not Found). Orchestrator response: \\r\\nCould not find the asset 'myHiddenAset'. Error code: 1002\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-10T14:32:13.412Z\",\r\n \"Message\": \"Could not find the asset 'myHiddenAset'. Error code: 1002\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "AssetLookupRunner" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetLookupRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-10T14:32:11.080Z\",\r\n \"EndTime\": \"2026-05-10T14:32:13.420Z\",\r\n \"CreationTime\": \"2026-05-10T14:32:10.825Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetLookupRunner\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-10T14:32:11.080Z\",\r\n \"EndTime\": \"2026-05-10T14:32:13.420Z\",\r\n \"CreationTime\": \"2026-05-10T14:32:10.825Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"StartTime\": \"2026-05-10T14:32:11.080Z\",\r\n \"EndTime\": \"2026-05-10T14:32:13.420Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"8c2f4e6a-1b3d-4f5a-9c2e-7d8f0a1b3c4d\",\r\n \"Info\": \"Get Credential: Status code: 404 (Not Found). Orchestrator response: Could not find the asset 'myHiddenAset'. Error code: 1002\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ServiceModel.FaultException: Could not find the asset 'myHiddenAset'. Error code: 1002\",\r\n \"CreationTime\": \"2026-05-10T14:32:10.825Z\",\r\n \"ReleaseName\": \"AssetLookupRunner\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5012840,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-10T14:32:13.420Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7f3e2a1b-9c8d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"Id\": 670920718\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5a8b9c2d-1e3f-4a6b-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/task.yaml index 9b8c197cc2..e6fd8fa86a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-name-mismatch/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/README.md new file mode 100644 index 0000000000..b6b73462c0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/README.md @@ -0,0 +1,86 @@ +# Get Asset Failure — Network / Connectivity Issue + +This scenario reproduces a runtime `Get Credential` failure caused by +the **robot host being unable to reach the Orchestrator URL** — +network, firewall, DNS, proxy, SSL/TLS, or session-expiry. The asset, +folder, robot license, and robot roles are all correctly configured; +the failing layer is below all of them, in the robot-to-Orchestrator +connectivity path. + +Orchestrator's response surfaces in the job logs as: + +``` +[Get Credential] Orchestrator information is not available. +Connection timed out connecting to https://cloud.uipath.com. +``` + +## What this scenario uncovers + +**Root Cause:** The robot host running this job (`RobotUser1` on +`MOCK-HOST`) cannot reach the Orchestrator URL. The CLI commands that +query Orchestrator from the developer's session (folders, jobs, +assets, users) all succeed because they run from a different network +context. Only the in-job HTTP call from the robot to Orchestrator +times out, surfacing as "Orchestrator information is not available." + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-network-connectivity.md` +(**low-confidence** playbook). + +> **Why "low-confidence":** the playbook's symptom — "Orchestrator +> information is not available" — covers many distinct sub-causes +> (Robot service not running, firewall, DNS, proxy, SSL cert, TLS +> EMS, session expiry). The agent's job here is to recognize the +> **connectivity layer** as the failing layer, not to pinpoint the +> exact sub-cause. The LLM judge is correspondingly flexible: any +> connectivity-family conclusion + connectivity-family fix scores +> high. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project, correctly configured at every workflow layer | + +The decisive evidence: + +1. **Job logs** contain "Orchestrator information is not available" along with connection timeout / TCP failure language. +2. **Asset list** shows `myHiddenAsset` present and correctly typed — rules out asset-side issues. +3. **`users list`** shows `RobotUser1` is licensed with the right roles — rules out license/permission issues. +4. **Folders list** returns `Remote Debugging` — rules out folder-scope issues. + +The CLI-query-success / robot-call-failure asymmetry is the signal: the developer's session can reach Orchestrator; the robot host cannot. + +## How this differs from sibling scenarios + +| Dimension | Other 8 GetAsset scenarios | `network-connectivity` (this) | +|---|---|---| +| Failing layer | Asset, folder, permission, type, per-robot value, license, vault, package | **Robot-to-Orchestrator connectivity** | +| Error signature | Codes 1002 / 1100 / 0 / 400 / 2304 / "Invalid asset type" / NRE | **"Orchestrator information is not available" + timeout/SSL** | +| Playbook confidence | high (5 scenarios) / medium (3) | **low** | +| Sub-cause variants in playbook | 1–3 per playbook | **6+** (Robot service, firewall, DNS, proxy, SSL, TLS EMS, session expiry) | +| Judge tolerance for sub-cause | strict (must name) | **flexible (family recognition is enough)** | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached a connectivity-family conclusion +- Conclusion must (a) name "Orchestrator information is not available" as the defining symptom and (b) recommend at least one connectivity-family fix (Robot service, network/firewall, proxy, SSL/TLS) + +The judge is permissive about which sub-cause the agent picks, since the playbook itself enumerates many. + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-network-connectivity --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/46d9acc45e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/46d9acc45e.json deleted file mode 100644 index b2617c53b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/46d9acc45e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/6945bfff10.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/6945bfff10.json deleted file mode 100644 index fdcb3e3765..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/6945bfff10.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f", - "State": "Faulted", - "ProcessName": "TenantConfigReader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T19:18:42.105Z", - "EndTime": "2026-05-13T19:19:13.812Z", - "CreationTime": "2026-05-13T19:18:41.864Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/703e06aea0.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/703e06aea0.json deleted file mode 100644 index cab362706a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/703e06aea0.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "UserList", - "Data": [ - { - "Id": 1042111, - "UserName": "user1", - "DisplayName": "user1", - "Type": "User", - "Roles": ["Administrator"], - "IsLicensed": true, - "LicenseType": "Administrator" - }, - { - "Id": 1042118, - "UserName": "RobotUser1", - "DisplayName": "Robot User 1", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"], - "IsLicensed": true, - "LicenseType": "Unattended", - "RobotProvision": { - "MachineName": "MOCK-HOST", - "ConnectionState": "Connected" - } - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/78de5bc4a5.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/78de5bc4a5.json deleted file mode 100644 index c41c1682b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/78de5bc4a5.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T19:19:13.781Z", - "Message": "[Get Credential] Orchestrator information is not available. Connection timed out connecting to https://cloud.uipath.com after 30000 ms.\r\nInner exception: System.Net.WebException: The operation has timed out." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T19:19:13.802Z", - "Message": "Orchestrator information is not available.\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/README.md deleted file mode 100644 index c5c1cb0242..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/README.md +++ /dev/null @@ -1,85 +0,0 @@ -# Get Asset Failure — Network / Connectivity Issue - -This scenario reproduces a runtime `Get Credential` failure caused by -the **robot host being unable to reach the Orchestrator URL** — -network, firewall, DNS, proxy, SSL/TLS, or session-expiry. The asset, -folder, robot license, and robot roles are all correctly configured; -the failing layer is below all of them, in the robot-to-Orchestrator -connectivity path. - -Orchestrator's response surfaces in the job logs as: - -``` -[Get Credential] Orchestrator information is not available. -Connection timed out connecting to https://cloud.uipath.com. -``` - -## What this scenario uncovers - -**Root Cause:** The robot host running this job (`RobotUser1` on -`MOCK-HOST`) cannot reach the Orchestrator URL. The CLI commands that -query Orchestrator from the developer's session (folders, jobs, -assets, users) all succeed because they run from a different network -context. Only the in-job HTTP call from the robot to Orchestrator -times out, surfacing as "Orchestrator information is not available." - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-network-connectivity.md` -(**low-confidence** playbook). - -> **Why "low-confidence":** the playbook's symptom — "Orchestrator -> information is not available" — covers many distinct sub-causes -> (Robot service not running, firewall, DNS, proxy, SSL cert, TLS -> EMS, session expiry). The agent's job here is to recognize the -> **connectivity layer** as the failing layer, not to pinpoint the -> exact sub-cause. The LLM judge is correspondingly flexible: any -> connectivity-family conclusion + connectivity-family fix scores -> high. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project, correctly configured at every workflow layer | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The decisive evidence: - -1. **Job logs** contain "Orchestrator information is not available" along with connection timeout / TCP failure language. -2. **Asset list** shows `myHiddenAsset` present and correctly typed — rules out asset-side issues. -3. **`users list`** shows `RobotUser1` is licensed with the right roles — rules out license/permission issues. -4. **Folders list** returns `Remote Debugging` — rules out folder-scope issues. - -The CLI-query-success / robot-call-failure asymmetry is the signal: the developer's session can reach Orchestrator; the robot host cannot. - -## How this differs from sibling scenarios - -| Dimension | Other 8 GetAsset scenarios | `network-connectivity` (this) | -|---|---|---| -| Failing layer | Asset, folder, permission, type, per-robot value, license, vault, package | **Robot-to-Orchestrator connectivity** | -| Error signature | Codes 1002 / 1100 / 0 / 400 / 2304 / "Invalid asset type" / NRE | **"Orchestrator information is not available" + timeout/SSL** | -| Playbook confidence | high (5 scenarios) / medium (3) | **low** | -| Sub-cause variants in playbook | 1–3 per playbook | **6+** (Robot service, firewall, DNS, proxy, SSL, TLS EMS, session expiry) | -| Judge tolerance for sub-cause | strict (must name) | **flexible (family recognition is enough)** | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached a connectivity-family conclusion -- Conclusion must (a) name "Orchestrator information is not available" as the defining symptom and (b) recommend at least one connectivity-family fix (Robot service, network/firewall, proxy, SSL/TLS) - -The judge is permissive about which sub-cause the agent picks, since the playbook itself enumerates many. - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-network-connectivity --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/ec4ad05227.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/ec4ad05227.json deleted file mode 100644 index dba4d14873..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/ec4ad05227.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f", - "StartTime": "2026-05-13T19:18:42.105Z", - "EndTime": "2026-05-13T19:19:13.812Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "3a5c7e9b-2d4f-4a6b-8c1d-9e2f3a4b5c6d", - "Info": "Get Credential: Orchestrator information is not available. Connection timed out connecting to https://cloud.uipath.com after 30000 ms.\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.Activities.OrchestratorCommunicationException: Orchestrator information is not available. Connection timed out connecting to https://cloud.uipath.com after 30000 ms.\r\n Inner exception: System.Net.WebException: The operation has timed out.", - "CreationTime": "2026-05-13T19:18:41.864Z", - "ReleaseName": "TenantConfigReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5043011, - "OrganizationUnitId": 7791626, - "FolderKey": "6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T19:19:13.812Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f", - "Id": 670985421 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/f7c9525731.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/f7c9525731.json deleted file mode 100644 index 62c1c10cc0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/f7c9525731.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/manifest.json deleted file mode 100644 index 605493d5c9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/m/r/manifest.json +++ /dev/null @@ -1,90 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-network-connectivity scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "f7c9525731.json" - }, - { - "match": "or folders list --output json", - "file": "f7c9525731.json" - }, - { - "match": "or jobs list --folder-key \"6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e\" --state Faulted --output json", - "file": "6945bfff10.json" - }, - { - "match": "or jobs list --folder-key 6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e --state Faulted --output json", - "file": "6945bfff10.json" - }, - { - "match": "or jobs list --process-name TenantConfigReader", - "file": "6945bfff10.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "6945bfff10.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "6945bfff10.json" - }, - { - "match": "or jobs list --state Faulted --process-name AssetNetworkConnectivity", - "file": "6945bfff10.json", - "_doc": "Order-tolerant companion: state-first flag order; folder-key-first and process-name-first shapes are covered by the neighboring rules." - }, - { - "match": "or jobs list --folder-key 6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e --state Faulted", - "file": "6945bfff10.json", - "_doc": "Order-tolerant companion without the trailing --output json, so flag-decorated variants still match." - }, - { - "match": "or jobs get \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\" --output json", - "file": "ec4ad05227.json" - }, - { - "match": "or jobs get 8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f --output json", - "file": "ec4ad05227.json" - }, - { - "match": "or jobs logs \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\" --level Error --output json", - "file": "78de5bc4a5.json" - }, - { - "match": "or jobs logs 8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f --level Error --output json", - "file": "78de5bc4a5.json" - }, - { - "match": "or assets list --folder-key 6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-key \"6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e\" --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "46d9acc45e.json" - }, - { - "match": "or users list --output json", - "file": "703e06aea0.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/uip-fixture.json new file mode 100644 index 0000000000..123d4bef49 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/data/uip-fixture.json @@ -0,0 +1,167 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TenantConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T19:18:42.105Z\",\r\n \"EndTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"CreationTime\": \"2026-05-13T19:18:41.864Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted", + "--process-name", + "AssetNetworkConnectivity" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TenantConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T19:18:42.105Z\",\r\n \"EndTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"CreationTime\": \"2026-05-13T19:18:41.864Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TenantConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T19:18:42.105Z\",\r\n \"EndTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"CreationTime\": \"2026-05-13T19:18:41.864Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T19:19:13.781Z\",\r\n \"Message\": \"[Get Credential] Orchestrator information is not available. Connection timed out connecting to https://cloud.uipath.com after 30000 ms.\\r\\nInner exception: System.Net.WebException: The operation has timed out.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T19:19:13.802Z\",\r\n \"Message\": \"Orchestrator information is not available.\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "TenantConfigReader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TenantConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T19:18:42.105Z\",\r\n \"EndTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"CreationTime\": \"2026-05-13T19:18:41.864Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TenantConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T19:18:42.105Z\",\r\n \"EndTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"CreationTime\": \"2026-05-13T19:18:41.864Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"StartTime\": \"2026-05-13T19:18:42.105Z\",\r\n \"EndTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"3a5c7e9b-2d4f-4a6b-8c1d-9e2f3a4b5c6d\",\r\n \"Info\": \"Get Credential: Orchestrator information is not available. Connection timed out connecting to https://cloud.uipath.com after 30000 ms.\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorCommunicationException: Orchestrator information is not available. Connection timed out connecting to https://cloud.uipath.com after 30000 ms.\\r\\n Inner exception: System.Net.WebException: The operation has timed out.\",\r\n \"CreationTime\": \"2026-05-13T19:18:41.864Z\",\r\n \"ReleaseName\": \"TenantConfigReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5043011,\r\n \"OrganizationUnitId\": 7791626,\r\n \"FolderKey\": \"6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T19:19:13.812Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8e3f4d5c-2b1a-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"Id\": 670985421\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"6b9d2c5e-3a4f-4b8c-9d1e-2f3a4b5c6d7e\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserList\",\r\n \"Data\": [\r\n {\r\n \"Id\": 1042111,\r\n \"UserName\": \"user1\",\r\n \"DisplayName\": \"user1\",\r\n \"Type\": \"User\",\r\n \"Roles\": [\"Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Administrator\"\r\n },\r\n {\r\n \"Id\": 1042118,\r\n \"UserName\": \"RobotUser1\",\r\n \"DisplayName\": \"Robot User 1\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Unattended\",\r\n \"RobotProvision\": {\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"ConnectionState\": \"Connected\"\r\n }\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/task.yaml index b7996da89d..c2f97af7e6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-network-connectivity/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/README.md new file mode 100644 index 0000000000..bdc6be05c0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/README.md @@ -0,0 +1,73 @@ +# Get Asset Failure — Per-Robot Asset Has No Value + +This scenario reproduces a runtime `Get Asset` failure caused by a +**per-robot asset configured without a value entry for the executing +robot**. Orchestrator returns "The asset 'MyPerRobotConfig' does not +have a value associated with this robot." + +## What this scenario uncovers + +**Root Cause:** The `Get Asset` activity in `Main.xaml` targets a +correctly-named asset in a real folder, and the asset's `ValueType` is +compatible with the activity. But the asset is configured with +`ValueScope: "PerRobot"` and the robot running the job has no entry in +the per-robot value table. Orchestrator therefore reports the asset +has no value for this robot. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-per-robot-no-value.md` +(the "Asset uses 'Per Robot' value mode but no entry exists for the +executing robot" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — uses `Get Asset` (`GetRobotAsset`) rather than `Get Credential` to match the playbook's coverage | + +> **Note on fixtures.** Like sibling scenarios, fixtures here were +> authored from the documented playbook signature rather than captured +> from a real `.local/investigations/` session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression signal. + +## How this differs from sibling scenarios + +| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` | `per-robot-no-value` (this) | +|---|---|---|---|---|---| +| Activity | Get Credential | Get Credential | Get Credential | Get Credential | **Get Asset** | +| `AssetName` | typo | correct | correct | correct | correct | +| `FolderPath` | real folder | non-existent | real | real | real | +| Asset visible in folder asset list? | no | n/a | yes | yes (but wrong type) | yes (right type) | +| `ValueScope` | n/a | n/a | n/a | n/a | **PerRobot** | +| Per-robot value entry for executing robot? | n/a | n/a | n/a | n/a | **no** | +| HTTP / error signature | 404 / 1002, "Could not find the asset" | 403 / 1100, "Folder ... does not exist" | 403 / 0, "You are not authorized" | 400, "Invalid asset type" | **"does not have a value associated with this robot"** | +| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | `get-asset-wrong-activity-type.md` | `get-asset-per-robot-no-value.md` | + +This scenario forces the agent past four rule-outs (asset exists, folder +exists, permission is fine, type matches) before landing on the +per-robot configuration. The asset list fixture is the decisive signal: +it shows `MyPerRobotConfig` with `ValueScope: "PerRobot"`, which together +with the error message points unambiguously at the per-robot value gap. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Conclusion must name the per-robot scope and recommend either adding a per-robot value entry OR switching the asset's scope to Global + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-per-robot-no-value --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/091e30f0a1.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/091e30f0a1.json deleted file mode 100644 index fd23f2ac4a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/091e30f0a1.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c", - "StartTime": "2026-05-13T11:08:24.180Z", - "EndTime": "2026-05-13T11:08:25.241Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "6e3f8a4b-2c5d-4f7e-9d3c-1a2b3c4d5e6f", - "Info": "Get Asset: The asset 'MyPerRobotConfig' does not have a value associated with this robot.\r\n\r\nin Main.xaml\r\n at GetRobotAsset \"Get Asset\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.Activities.OrchestratorCommunicationException: The asset 'MyPerRobotConfig' does not have a value associated with this robot.", - "CreationTime": "2026-05-13T11:08:23.950Z", - "ReleaseName": "PerRobotConfigReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5028139, - "OrganizationUnitId": 7791622, - "FolderKey": "2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T11:08:25.241Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c", - "Id": 670961094 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/1c0fece232.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/1c0fece232.json deleted file mode 100644 index 10d073589c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/1c0fece232.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "d4e5f6a7-b8c9-0123-defa-1234567890bc", - "Name": "MyPerRobotConfig", - "ValueType": "Text", - "ValueScope": "PerRobot", - "Description": "Per-robot configuration value. Each robot is expected to have its own entry in the per-robot value table." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/README.md deleted file mode 100644 index 50045447a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/README.md +++ /dev/null @@ -1,72 +0,0 @@ -# Get Asset Failure — Per-Robot Asset Has No Value - -This scenario reproduces a runtime `Get Asset` failure caused by a -**per-robot asset configured without a value entry for the executing -robot**. Orchestrator returns "The asset 'MyPerRobotConfig' does not -have a value associated with this robot." - -## What this scenario uncovers - -**Root Cause:** The `Get Asset` activity in `Main.xaml` targets a -correctly-named asset in a real folder, and the asset's `ValueType` is -compatible with the activity. But the asset is configured with -`ValueScope: "PerRobot"` and the robot running the job has no entry in -the per-robot value table. Orchestrator therefore reports the asset -has no value for this robot. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-per-robot-no-value.md` -(the "Asset uses 'Per Robot' value mode but no entry exists for the -executing robot" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | synthesized UiPath project — uses `Get Asset` (`GetRobotAsset`) rather than `Get Credential` to match the playbook's coverage | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like sibling scenarios, fixtures here were -> authored from the documented playbook signature rather than captured -> from a real `.local/investigations/` session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression signal. - -## How this differs from sibling scenarios - -| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` | `per-robot-no-value` (this) | -|---|---|---|---|---|---| -| Activity | Get Credential | Get Credential | Get Credential | Get Credential | **Get Asset** | -| `AssetName` | typo | correct | correct | correct | correct | -| `FolderPath` | real folder | non-existent | real | real | real | -| Asset visible in folder asset list? | no | n/a | yes | yes (but wrong type) | yes (right type) | -| `ValueScope` | n/a | n/a | n/a | n/a | **PerRobot** | -| Per-robot value entry for executing robot? | n/a | n/a | n/a | n/a | **no** | -| HTTP / error signature | 404 / 1002, "Could not find the asset" | 403 / 1100, "Folder ... does not exist" | 403 / 0, "You are not authorized" | 400, "Invalid asset type" | **"does not have a value associated with this robot"** | -| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | `get-asset-wrong-activity-type.md` | `get-asset-per-robot-no-value.md` | - -This scenario forces the agent past four rule-outs (asset exists, folder -exists, permission is fine, type matches) before landing on the -per-robot configuration. The asset list fixture is the decisive signal: -it shows `MyPerRobotConfig` with `ValueScope: "PerRobot"`, which together -with the error message points unambiguously at the per-robot value gap. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Conclusion must name the per-robot scope and recommend either adding a per-robot value entry OR switching the asset's scope to Global - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-per-robot-no-value --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/a77a9c9516.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/a77a9c9516.json deleted file mode 100644 index 17bf839ead..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/a77a9c9516.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T11:08:25.215Z", - "Message": "[Get Asset] Orchestrator response: \r\nThe asset 'MyPerRobotConfig' does not have a value associated with this robot." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T11:08:25.232Z", - "Message": "The asset 'MyPerRobotConfig' does not have a value associated with this robot.\r\nin Main.xaml\r\n at GetRobotAsset \"Get Asset\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/c0f70a5cf5.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/c0f70a5cf5.json deleted file mode 100644 index b05d1223c8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/c0f70a5cf5.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c", - "State": "Faulted", - "ProcessName": "PerRobotConfigReader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T11:08:24.180Z", - "EndTime": "2026-05-13T11:08:25.241Z", - "CreationTime": "2026-05-13T11:08:23.950Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/c2f12ac5da.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/c2f12ac5da.json deleted file mode 100644 index 5974b64443..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/c2f12ac5da.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/e3632053a6.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/e3632053a6.json deleted file mode 100644 index dc4cbd4507..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/e3632053a6.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "UserList", - "Data": [ - { - "Id": 1042111, - "UserName": "user1", - "DisplayName": "user1", - "Type": "User", - "Roles": ["Administrator"] - }, - { - "Id": 1042118, - "UserName": "RobotUser1", - "DisplayName": "Robot User 1", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"] - }, - { - "Id": 1042119, - "UserName": "RobotUser2", - "DisplayName": "Robot User 2", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"] - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/manifest.json deleted file mode 100644 index ffb2723f1b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/m/r/manifest.json +++ /dev/null @@ -1,80 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-per-robot-no-value scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "c2f12ac5da.json" - }, - { - "match": "or folders list --output json", - "file": "c2f12ac5da.json" - }, - { - "match": "or jobs list --folder-key \"2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f\" --state Faulted --output json", - "file": "c0f70a5cf5.json" - }, - { - "match": "or jobs list --folder-key 2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f --state Faulted --output json", - "file": "c0f70a5cf5.json" - }, - { - "match": "or jobs list --process-name PerRobotConfigReader", - "file": "c0f70a5cf5.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "c0f70a5cf5.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "c0f70a5cf5.json" - }, - { - "match": "or jobs get \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\" --output json", - "file": "091e30f0a1.json" - }, - { - "match": "or jobs get 4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c --output json", - "file": "091e30f0a1.json" - }, - { - "match": "or jobs logs \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\" --level Error --output json", - "file": "a77a9c9516.json" - }, - { - "match": "or jobs logs 4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c --level Error --output json", - "file": "a77a9c9516.json" - }, - { - "match": "or assets list --folder-key 2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f --output json", - "file": "1c0fece232.json" - }, - { - "match": "or assets list --folder-key \"2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f\" --output json", - "file": "1c0fece232.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "1c0fece232.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "1c0fece232.json" - }, - { - "match": "or users list --output json", - "file": "e3632053a6.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/uip-fixture.json new file mode 100644 index 0000000000..16ff804fa6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PerRobotConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:08:24.180Z\",\r\n \"EndTime\": \"2026-05-13T11:08:25.241Z\",\r\n \"CreationTime\": \"2026-05-13T11:08:23.950Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PerRobotConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:08:24.180Z\",\r\n \"EndTime\": \"2026-05-13T11:08:25.241Z\",\r\n \"CreationTime\": \"2026-05-13T11:08:23.950Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T11:08:25.215Z\",\r\n \"Message\": \"[Get Asset] Orchestrator response: \\r\\nThe asset 'MyPerRobotConfig' does not have a value associated with this robot.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T11:08:25.232Z\",\r\n \"Message\": \"The asset 'MyPerRobotConfig' does not have a value associated with this robot.\\r\\nin Main.xaml\\r\\n at GetRobotAsset \\\"Get Asset\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-0123-defa-1234567890bc\",\r\n \"Name\": \"MyPerRobotConfig\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"PerRobot\",\r\n \"Description\": \"Per-robot configuration value. Each robot is expected to have its own entry in the per-robot value table.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "PerRobotConfigReader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PerRobotConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:08:24.180Z\",\r\n \"EndTime\": \"2026-05-13T11:08:25.241Z\",\r\n \"CreationTime\": \"2026-05-13T11:08:23.950Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PerRobotConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T11:08:24.180Z\",\r\n \"EndTime\": \"2026-05-13T11:08:25.241Z\",\r\n \"CreationTime\": \"2026-05-13T11:08:23.950Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-0123-defa-1234567890bc\",\r\n \"Name\": \"MyPerRobotConfig\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"PerRobot\",\r\n \"Description\": \"Per-robot configuration value. Each robot is expected to have its own entry in the per-robot value table.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-0123-defa-1234567890bc\",\r\n \"Name\": \"MyPerRobotConfig\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"PerRobot\",\r\n \"Description\": \"Per-robot configuration value. Each robot is expected to have its own entry in the per-robot value table.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\",\r\n \"StartTime\": \"2026-05-13T11:08:24.180Z\",\r\n \"EndTime\": \"2026-05-13T11:08:25.241Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"6e3f8a4b-2c5d-4f7e-9d3c-1a2b3c4d5e6f\",\r\n \"Info\": \"Get Asset: The asset 'MyPerRobotConfig' does not have a value associated with this robot.\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotAsset \\\"Get Asset\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorCommunicationException: The asset 'MyPerRobotConfig' does not have a value associated with this robot.\",\r\n \"CreationTime\": \"2026-05-13T11:08:23.950Z\",\r\n \"ReleaseName\": \"PerRobotConfigReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5028139,\r\n \"OrganizationUnitId\": 7791622,\r\n \"FolderKey\": \"2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T11:08:25.241Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"4d7e9f3a-1b6c-5e8f-9d4c-3b2a1f0e9d8c\",\r\n \"Id\": 670961094\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2c5e8f4a-9b3d-4f6c-8e1d-7a2b3c4d5e6f\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserList\",\r\n \"Data\": [\r\n {\r\n \"Id\": 1042111,\r\n \"UserName\": \"user1\",\r\n \"DisplayName\": \"user1\",\r\n \"Type\": \"User\",\r\n \"Roles\": [\"Administrator\"]\r\n },\r\n {\r\n \"Id\": 1042118,\r\n \"UserName\": \"RobotUser1\",\r\n \"DisplayName\": \"Robot User 1\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"]\r\n },\r\n {\r\n \"Id\": 1042119,\r\n \"UserName\": \"RobotUser2\",\r\n \"DisplayName\": \"Robot User 2\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"]\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/task.yaml index a3c4585931..324f6aa4da 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-per-robot-no-value/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/README.md new file mode 100644 index 0000000000..4ff42f751f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/README.md @@ -0,0 +1,70 @@ +# Get Asset Failure — Permission Denied + +This scenario reproduces a runtime `Get Credential` failure caused by +the **robot account lacking `Assets.View` permission** on the target +folder. Orchestrator returns HTTP **403** / error code **0** with the +message "You are not authorized! The robot account does not have the +required permissions on Assets in this folder." + +## What this scenario uncovers + +**Root Cause:** The `Get Credential` activity in `Main.xaml` references +a correctly-spelled asset (`myHiddenAsset`) in a real folder (`Remote +Debugging`). Both the asset and the folder exist and are visible to +the CLI user — but the robot account that runs the job lacks the +`Assets.View` permission on the folder, so Orchestrator returns 403. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-permission-denied.md` +(the "Robot account role does not include View permission on Assets" +branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — same scaffold as `getasset-name-mismatch` with the typo fixed and the FolderPath set to a real folder | + +> **Note on fixtures.** Like sibling scenarios, fixtures here were +> authored from the documented playbook signature rather than captured +> from a real `.local/investigations/` session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression signal. + +## How this differs from sibling scenarios + +| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` (this) | +|---|---|---|---| +| AssetName | typo (`myHiddenAset`) | correct (`myHiddenAsset`) | correct (`myHiddenAsset`) | +| FolderPath | real folder | non-existent (`OldDevFolder`) | real (`Remote Debugging`) | +| Asset visible in folder asset list? | no (different name) | n/a (folder doesn't exist) | **yes** | +| Folder visible in folder list? | yes | **no** | yes | +| HTTP status / error code | 404 / 1002 | 403 / 1100 | **403 / 0** | +| Error message anchor | "Could not find the asset" | "Folder ... does not exist or the user does not have access" | "You are not authorized" / "required permissions on Assets" | +| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | + +The asset list and folder list fixtures here both return the expected +positive content (asset is present, folder is present). The agent must +notice that what's failing is **authorization**, not **discovery**. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Conclusion must explicitly name the missing `Assets.View` permission (or equivalent role-level fix) on the robot account; must NOT reach the wrong conclusion as asset-not-found or folder-not-found + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-permission-denied --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/46d9acc45e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/46d9acc45e.json deleted file mode 100644 index b2617c53b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/46d9acc45e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/526c1dc17e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/526c1dc17e.json deleted file mode 100644 index 2501c9abc0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/526c1dc17e.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T15:42:20.291Z", - "Message": "[Get Credential] Status code: 403 (Forbidden). Orchestrator response: \r\nYou are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T15:42:20.307Z", - "Message": "You are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/README.md deleted file mode 100644 index 090a0632ba..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Get Asset Failure — Permission Denied - -This scenario reproduces a runtime `Get Credential` failure caused by -the **robot account lacking `Assets.View` permission** on the target -folder. Orchestrator returns HTTP **403** / error code **0** with the -message "You are not authorized! The robot account does not have the -required permissions on Assets in this folder." - -## What this scenario uncovers - -**Root Cause:** The `Get Credential` activity in `Main.xaml` references -a correctly-spelled asset (`myHiddenAsset`) in a real folder (`Remote -Debugging`). Both the asset and the folder exist and are visible to -the CLI user — but the robot account that runs the job lacks the -`Assets.View` permission on the folder, so Orchestrator returns 403. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-permission-denied.md` -(the "Robot account role does not include View permission on Assets" -branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | synthesized UiPath project — same scaffold as `getasset-name-mismatch` with the typo fixed and the FolderPath set to a real folder | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like sibling scenarios, fixtures here were -> authored from the documented playbook signature rather than captured -> from a real `.local/investigations/` session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression signal. - -## How this differs from sibling scenarios - -| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` (this) | -|---|---|---|---| -| AssetName | typo (`myHiddenAset`) | correct (`myHiddenAsset`) | correct (`myHiddenAsset`) | -| FolderPath | real folder | non-existent (`OldDevFolder`) | real (`Remote Debugging`) | -| Asset visible in folder asset list? | no (different name) | n/a (folder doesn't exist) | **yes** | -| Folder visible in folder list? | yes | **no** | yes | -| HTTP status / error code | 404 / 1002 | 403 / 1100 | **403 / 0** | -| Error message anchor | "Could not find the asset" | "Folder ... does not exist or the user does not have access" | "You are not authorized" / "required permissions on Assets" | -| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | - -The asset list and folder list fixtures here both return the expected -positive content (asset is present, folder is present). The agent must -notice that what's failing is **authorization**, not **discovery**. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Conclusion must explicitly name the missing `Assets.View` permission (or equivalent role-level fix) on the robot account; must NOT reach the wrong conclusion as asset-not-found or folder-not-found - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-permission-denied --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/b08a6b2edb.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/b08a6b2edb.json deleted file mode 100644 index 3b940cf50d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/b08a6b2edb.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/b6aed3308b.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/b6aed3308b.json deleted file mode 100644 index 06b7042940..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/b6aed3308b.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c", - "StartTime": "2026-05-12T15:42:18.220Z", - "EndTime": "2026-05-12T15:42:20.314Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7a2b4c5d-3e6f-4a7b-9c8d-0e1f2a3b4c5d", - "Info": "Get Credential: Status code: 403 (Forbidden). Orchestrator response: You are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.Activities.OrchestratorCommunicationException: You are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0", - "CreationTime": "2026-05-12T15:42:17.985Z", - "ReleaseName": "CredentialAssetLoader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5021744, - "OrganizationUnitId": 7791620, - "FolderKey": "7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T15:42:20.314Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c", - "Id": 670941885 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/f4d26b8984.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/f4d26b8984.json deleted file mode 100644 index fa7bf4e53d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/f4d26b8984.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "UserList", - "Data": [ - { - "Id": 1042111, - "UserName": "user1", - "DisplayName": "user1", - "Type": "User", - "Roles": ["Administrator"] - }, - { - "Id": 1042118, - "UserName": "RobotUser1", - "DisplayName": "Robot User 1", - "Type": "Robot", - "Roles": ["Robot"] - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/ffbc003668.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/ffbc003668.json deleted file mode 100644 index 754c05ad31..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/ffbc003668.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c", - "State": "Faulted", - "ProcessName": "CredentialAssetLoader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T15:42:18.220Z", - "EndTime": "2026-05-12T15:42:20.314Z", - "CreationTime": "2026-05-12T15:42:17.985Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/manifest.json deleted file mode 100644 index 9377d8ee83..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/m/r/manifest.json +++ /dev/null @@ -1,80 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-permission-denied scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "b08a6b2edb.json" - }, - { - "match": "or folders list --output json", - "file": "b08a6b2edb.json" - }, - { - "match": "or jobs list --folder-key \"7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a\" --state Faulted --output json", - "file": "ffbc003668.json" - }, - { - "match": "or jobs list --folder-key 7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a --state Faulted --output json", - "file": "ffbc003668.json" - }, - { - "match": "or jobs list --process-name CredentialAssetLoader", - "file": "ffbc003668.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "ffbc003668.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "ffbc003668.json" - }, - { - "match": "or jobs get \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\" --output json", - "file": "b6aed3308b.json" - }, - { - "match": "or jobs get 9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c --output json", - "file": "b6aed3308b.json" - }, - { - "match": "or jobs logs \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\" --level Error --output json", - "file": "526c1dc17e.json" - }, - { - "match": "or jobs logs 9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c --level Error --output json", - "file": "526c1dc17e.json" - }, - { - "match": "or assets list --folder-key 7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-key \"7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a\" --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "46d9acc45e.json" - }, - { - "match": "or users list --output json", - "file": "f4d26b8984.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/uip-fixture.json new file mode 100644 index 0000000000..521f76d374 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CredentialAssetLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T15:42:18.220Z\",\r\n \"EndTime\": \"2026-05-12T15:42:20.314Z\",\r\n \"CreationTime\": \"2026-05-12T15:42:17.985Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CredentialAssetLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T15:42:18.220Z\",\r\n \"EndTime\": \"2026-05-12T15:42:20.314Z\",\r\n \"CreationTime\": \"2026-05-12T15:42:17.985Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T15:42:20.291Z\",\r\n \"Message\": \"[Get Credential] Status code: 403 (Forbidden). Orchestrator response: \\r\\nYou are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T15:42:20.307Z\",\r\n \"Message\": \"You are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "CredentialAssetLoader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CredentialAssetLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T15:42:18.220Z\",\r\n \"EndTime\": \"2026-05-12T15:42:20.314Z\",\r\n \"CreationTime\": \"2026-05-12T15:42:17.985Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CredentialAssetLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T15:42:18.220Z\",\r\n \"EndTime\": \"2026-05-12T15:42:20.314Z\",\r\n \"CreationTime\": \"2026-05-12T15:42:17.985Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\",\r\n \"StartTime\": \"2026-05-12T15:42:18.220Z\",\r\n \"EndTime\": \"2026-05-12T15:42:20.314Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7a2b4c5d-3e6f-4a7b-9c8d-0e1f2a3b4c5d\",\r\n \"Info\": \"Get Credential: Status code: 403 (Forbidden). Orchestrator response: You are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorCommunicationException: You are not authorized! The robot account does not have the required permissions on Assets in this folder. Error code: 0\",\r\n \"CreationTime\": \"2026-05-12T15:42:17.985Z\",\r\n \"ReleaseName\": \"CredentialAssetLoader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5021744,\r\n \"OrganizationUnitId\": 7791620,\r\n \"FolderKey\": \"7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-12T15:42:20.314Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"9f5e4d3c-2b1a-4f8e-7d6c-5b4a3e2d1f0c\",\r\n \"Id\": 670941885\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7e9d4f2a-3b5c-4d8e-9f1a-2b3c4d5e6f7a\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserList\",\r\n \"Data\": [\r\n {\r\n \"Id\": 1042111,\r\n \"UserName\": \"user1\",\r\n \"DisplayName\": \"user1\",\r\n \"Type\": \"User\",\r\n \"Roles\": [\"Administrator\"]\r\n },\r\n {\r\n \"Id\": 1042118,\r\n \"UserName\": \"RobotUser1\",\r\n \"DisplayName\": \"Robot User 1\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\"]\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/task.yaml index 600b830f18..5c681e1127 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-permission-denied/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/README.md new file mode 100644 index 0000000000..3f5f2bce4f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/README.md @@ -0,0 +1,94 @@ +# Get Asset Failure — Robot Not Authenticated (API-Token Layer) + +This scenario reproduces a runtime `Get Credential` failure where the +robot is **licensed, connected, and correctly permissioned**, the asset +and folder are correctly configured — yet the activity's Orchestrator +REST call is rejected with HTTP **401** / "You are not authenticated! +Error code: 0". The failure is at the robot **identity / API-token** +layer, surfaced *after* the job has already started running. + +## What this scenario uncovers + +**Root Cause:** The Get Credential activity in `Main.xaml` targets a +correctly-named Credential asset (`myHiddenAsset`) in a real folder +(`Remote Debugging`). All asset-level configuration is correct, the +robot account (`RobotUser1`) holds the right roles, and it is licensed +and connected (`IsLicensed: true`, `LicenseType: Unattended`). The job +**started and ran ~1 second** before faulting *inside* `Main.xaml` at +`GetRobotCredential` — which proves the robot acquired a runtime, so +licensing is not the problem. The robot's Orchestrator API token is +being rejected when the activity calls the REST API. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-robot-not-authenticated.md` +(medium-confidence playbook). + +> **Why "medium-confidence":** the same 401 / "not authenticated" +> signature on a *running* job can arise from several robot-identity +> causes per the playbook — machine-key / client-credential mismatch, +> robot-key authentication disabled in tenant security, or a +> `UiPath.System.Activities` auth regression. The available evidence +> (everything else correct) does not single one out, so the resolution +> is a ranked checklist. The agent must localize the fault to the +> authentication / token layer and rule out the licensing red herring. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — Get Credential activity, correctly configured at every asset layer | + +The smoking gun is the combination of: + +1. The error message phrasing — "not **authenticated**" / HTTP 401 + (distinct from permission-denied's "not **authorized**" / 403). +2. The job has a `StartTime` and ran ~1s before faulting at the + activity — it **executed**, so the robot is licensed enough to run. +3. `or users list` shows `RobotUser1` licensed and connected + (`IsLicensed: true`, `LicenseType: Unattended`, + `ConnectionState: Connected`), and asset / folder / type / + permissions all check out — leaving the robot's API-token + authentication as the only failing layer. + +## How this differs from sibling scenarios + +| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` | `per-robot-no-value` | `robot-not-authenticated` (this) | +|---|---|---|---|---|---|---| +| AssetName | typo | correct | correct | correct | correct | correct | +| FolderPath | real | non-existent | real | real | real | real | +| Asset visible in folder asset list? | no | n/a | yes | yes (wrong type) | yes (right type) | yes | +| Permission issue? | n/a | n/a | yes (`Assets.View` missing) | n/a | n/a | no (robot has Assets.View) | +| Robot licensed & connected? | n/a | n/a | n/a | n/a | n/a | **yes — licensed, connected** | +| Did the job start/run? | n/a | n/a | n/a | n/a | n/a | **yes — ran ~1s, faulted at the activity** | +| Error phrasing | "Could not find the asset" | "Folder ... does not exist" | "not **authorized**" | "Invalid asset type" | "does not have a value associated with this robot" | **"not authenticated"** | +| HTTP / error code | 404 / 1002 | 403 / 1100 | 403 / 0 | 400 | n/a | **401 / 0** | +| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | `get-asset-wrong-activity-type.md` | `get-asset-per-robot-no-value.md` | `get-asset-robot-not-authenticated.md` | + +The agent must distinguish "not authenticated" (401, this scenario) +from "not authorized" (403, `permission-denied`) — same error code (0) +but different word, different cause, different fix — AND avoid the +licensing red herring: an unlicensed robot would never have started the +job, so a *running* job that 401s is an authentication/token failure, +not a licensing gap. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Conclusion must (a) name the robot account `RobotUser1`, (b) localize the failure to the robot authentication / API-token layer (explicitly NOT licensing, explicitly NOT permission), and (c) recommend at least one of: reconnect with the correct machine key, enable robot-key authentication, or check `UiPath.System.Activities` for the auth regression + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-robot-not-authenticated --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/2f14875026.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/2f14875026.json deleted file mode 100644 index 2a6db7f57a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/2f14875026.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "UserList", - "Data": [ - { - "Id": 1042111, - "UserName": "user1", - "DisplayName": "user1", - "Type": "User", - "Roles": ["Administrator"], - "IsLicensed": true, - "LicenseType": "Administrator" - }, - { - "Id": 1042118, - "UserName": "RobotUser1", - "DisplayName": "Robot User 1", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"], - "IsLicensed": true, - "LicenseType": "Unattended", - "RobotProvision": { - "MachineName": "MOCK-HOST", - "ConnectionState": "Connected" - } - }, - { - "Id": 1042119, - "UserName": "RobotUser2", - "DisplayName": "Robot User 2", - "Type": "Robot", - "Roles": ["Robot", "Asset Administrator"], - "IsLicensed": true, - "LicenseType": "Unattended", - "RobotProvision": { - "MachineName": "MOCK-HOST", - "ConnectionState": "Connected" - } - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/46d9acc45e.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/46d9acc45e.json deleted file mode 100644 index b2617c53b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/46d9acc45e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/5894eac7f1.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/5894eac7f1.json deleted file mode 100644 index 73a1648e66..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/5894eac7f1.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d", - "State": "Faulted", - "ProcessName": "RuntimeConfigReader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T13:24:17.510Z", - "EndTime": "2026-05-13T13:24:18.602Z", - "CreationTime": "2026-05-13T13:24:17.285Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/611400a545.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/611400a545.json deleted file mode 100644 index fe1b35b5ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/611400a545.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/93f4239e9d.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/93f4239e9d.json deleted file mode 100644 index eac75a8edf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/93f4239e9d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d", - "StartTime": "2026-05-13T13:24:17.510Z", - "EndTime": "2026-05-13T13:24:18.602Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "8d2f4e6a-5b1c-4a3e-9d2f-7c8b1a2d3e4f", - "Info": "Get Credential: Status code: 401 (Unauthorized). Orchestrator response: You are not authenticated! Error code: 0\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.Activities.OrchestratorCommunicationException: You are not authenticated! Error code: 0", - "CreationTime": "2026-05-13T13:24:17.285Z", - "ReleaseName": "RuntimeConfigReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5031466, - "OrganizationUnitId": 7791623, - "FolderKey": "3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ROBOTUSER1", - "OrchestratorUserIdentity": "RobotUser1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T13:24:18.602Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d", - "Id": 670967853 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/README.md deleted file mode 100644 index cbf002dbc9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/README.md +++ /dev/null @@ -1,93 +0,0 @@ -# Get Asset Failure — Robot Not Authenticated (API-Token Layer) - -This scenario reproduces a runtime `Get Credential` failure where the -robot is **licensed, connected, and correctly permissioned**, the asset -and folder are correctly configured — yet the activity's Orchestrator -REST call is rejected with HTTP **401** / "You are not authenticated! -Error code: 0". The failure is at the robot **identity / API-token** -layer, surfaced *after* the job has already started running. - -## What this scenario uncovers - -**Root Cause:** The Get Credential activity in `Main.xaml` targets a -correctly-named Credential asset (`myHiddenAsset`) in a real folder -(`Remote Debugging`). All asset-level configuration is correct, the -robot account (`RobotUser1`) holds the right roles, and it is licensed -and connected (`IsLicensed: true`, `LicenseType: Unattended`). The job -**started and ran ~1 second** before faulting *inside* `Main.xaml` at -`GetRobotCredential` — which proves the robot acquired a runtime, so -licensing is not the problem. The robot's Orchestrator API token is -being rejected when the activity calls the REST API. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-robot-not-authenticated.md` -(medium-confidence playbook). - -> **Why "medium-confidence":** the same 401 / "not authenticated" -> signature on a *running* job can arise from several robot-identity -> causes per the playbook — machine-key / client-credential mismatch, -> robot-key authentication disabled in tenant security, or a -> `UiPath.System.Activities` auth regression. The available evidence -> (everything else correct) does not single one out, so the resolution -> is a ranked checklist. The agent must localize the fault to the -> authentication / token layer and rule out the licensing red herring. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | synthesized UiPath project — Get Credential activity, correctly configured at every asset layer | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The smoking gun is the combination of: - -1. The error message phrasing — "not **authenticated**" / HTTP 401 - (distinct from permission-denied's "not **authorized**" / 403). -2. The job has a `StartTime` and ran ~1s before faulting at the - activity — it **executed**, so the robot is licensed enough to run. -3. `or users list` shows `RobotUser1` licensed and connected - (`IsLicensed: true`, `LicenseType: Unattended`, - `ConnectionState: Connected`), and asset / folder / type / - permissions all check out — leaving the robot's API-token - authentication as the only failing layer. - -## How this differs from sibling scenarios - -| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` | `per-robot-no-value` | `robot-not-authenticated` (this) | -|---|---|---|---|---|---|---| -| AssetName | typo | correct | correct | correct | correct | correct | -| FolderPath | real | non-existent | real | real | real | real | -| Asset visible in folder asset list? | no | n/a | yes | yes (wrong type) | yes (right type) | yes | -| Permission issue? | n/a | n/a | yes (`Assets.View` missing) | n/a | n/a | no (robot has Assets.View) | -| Robot licensed & connected? | n/a | n/a | n/a | n/a | n/a | **yes — licensed, connected** | -| Did the job start/run? | n/a | n/a | n/a | n/a | n/a | **yes — ran ~1s, faulted at the activity** | -| Error phrasing | "Could not find the asset" | "Folder ... does not exist" | "not **authorized**" | "Invalid asset type" | "does not have a value associated with this robot" | **"not authenticated"** | -| HTTP / error code | 404 / 1002 | 403 / 1100 | 403 / 0 | 400 | n/a | **401 / 0** | -| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | `get-asset-wrong-activity-type.md` | `get-asset-per-robot-no-value.md` | `get-asset-robot-not-authenticated.md` | - -The agent must distinguish "not authenticated" (401, this scenario) -from "not authorized" (403, `permission-denied`) — same error code (0) -but different word, different cause, different fix — AND avoid the -licensing red herring: an unlicensed robot would never have started the -job, so a *running* job that 401s is an authentication/token failure, -not a licensing gap. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Conclusion must (a) name the robot account `RobotUser1`, (b) localize the failure to the robot authentication / API-token layer (explicitly NOT licensing, explicitly NOT permission), and (c) recommend at least one of: reconnect with the correct machine key, enable robot-key authentication, or check `UiPath.System.Activities` for the auth regression - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-robot-not-authenticated --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/bc11beea18.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/bc11beea18.json deleted file mode 100644 index 7654a734dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/bc11beea18.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T13:24:18.580Z", - "Message": "[Get Credential] Status code: 401 (Unauthorized). Orchestrator response: \r\nYou are not authenticated! Error code: 0" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T13:24:18.594Z", - "Message": "You are not authenticated! Error code: 0\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/manifest.json deleted file mode 100644 index 064ef76100..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/m/r/manifest.json +++ /dev/null @@ -1,80 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-robot-not-authenticated scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "611400a545.json" - }, - { - "match": "or folders list --output json", - "file": "611400a545.json" - }, - { - "match": "or jobs list --folder-key \"3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f\" --state Faulted --output json", - "file": "5894eac7f1.json" - }, - { - "match": "or jobs list --folder-key 3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f --state Faulted --output json", - "file": "5894eac7f1.json" - }, - { - "match": "or jobs list --process-name RobotConfigLoader", - "file": "5894eac7f1.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "5894eac7f1.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "5894eac7f1.json" - }, - { - "match": "or jobs get \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\" --output json", - "file": "93f4239e9d.json" - }, - { - "match": "or jobs get 5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d --output json", - "file": "93f4239e9d.json" - }, - { - "match": "or jobs logs \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\" --level Error --output json", - "file": "bc11beea18.json" - }, - { - "match": "or jobs logs 5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d --level Error --output json", - "file": "bc11beea18.json" - }, - { - "match": "or assets list --folder-key 3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-key \"3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f\" --output json", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "46d9acc45e.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "46d9acc45e.json" - }, - { - "match": "or users list --output json", - "file": "2f14875026.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/uip-fixture.json new file mode 100644 index 0000000000..e5f68a359f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RuntimeConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T13:24:17.510Z\",\r\n \"EndTime\": \"2026-05-13T13:24:18.602Z\",\r\n \"CreationTime\": \"2026-05-13T13:24:17.285Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RuntimeConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T13:24:17.510Z\",\r\n \"EndTime\": \"2026-05-13T13:24:18.602Z\",\r\n \"CreationTime\": \"2026-05-13T13:24:17.285Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T13:24:18.580Z\",\r\n \"Message\": \"[Get Credential] Status code: 401 (Unauthorized). Orchestrator response: \\r\\nYou are not authenticated! Error code: 0\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T13:24:18.594Z\",\r\n \"Message\": \"You are not authenticated! Error code: 0\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "RobotConfigLoader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RuntimeConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T13:24:17.510Z\",\r\n \"EndTime\": \"2026-05-13T13:24:18.602Z\",\r\n \"CreationTime\": \"2026-05-13T13:24:17.285Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RuntimeConfigReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T13:24:17.510Z\",\r\n \"EndTime\": \"2026-05-13T13:24:18.602Z\",\r\n \"CreationTime\": \"2026-05-13T13:24:17.285Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\",\r\n \"StartTime\": \"2026-05-13T13:24:17.510Z\",\r\n \"EndTime\": \"2026-05-13T13:24:18.602Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"8d2f4e6a-5b1c-4a3e-9d2f-7c8b1a2d3e4f\",\r\n \"Info\": \"Get Credential: Status code: 401 (Unauthorized). Orchestrator response: You are not authenticated! Error code: 0\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorCommunicationException: You are not authenticated! Error code: 0\",\r\n \"CreationTime\": \"2026-05-13T13:24:17.285Z\",\r\n \"ReleaseName\": \"RuntimeConfigReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5031466,\r\n \"OrganizationUnitId\": 7791623,\r\n \"FolderKey\": \"3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ROBOTUSER1\",\r\n \"OrchestratorUserIdentity\": \"RobotUser1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T13:24:18.602Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"5b8c4d2a-9e3f-4a6b-7c8d-1e2f3a4b5c6d\",\r\n \"Id\": 670967853\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3e6f9b4c-1a2d-4f7e-8c5d-9b3a4c5d6e7f\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserList\",\r\n \"Data\": [\r\n {\r\n \"Id\": 1042111,\r\n \"UserName\": \"user1\",\r\n \"DisplayName\": \"user1\",\r\n \"Type\": \"User\",\r\n \"Roles\": [\"Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Administrator\"\r\n },\r\n {\r\n \"Id\": 1042118,\r\n \"UserName\": \"RobotUser1\",\r\n \"DisplayName\": \"Robot User 1\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Unattended\",\r\n \"RobotProvision\": {\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"ConnectionState\": \"Connected\"\r\n }\r\n },\r\n {\r\n \"Id\": 1042119,\r\n \"UserName\": \"RobotUser2\",\r\n \"DisplayName\": \"Robot User 2\",\r\n \"Type\": \"Robot\",\r\n \"Roles\": [\"Robot\", \"Asset Administrator\"],\r\n \"IsLicensed\": true,\r\n \"LicenseType\": \"Unattended\",\r\n \"RobotProvision\": {\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"ConnectionState\": \"Connected\"\r\n }\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/task.yaml index 4f585c2f0c..220210d8bb 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-robot-not-authenticated/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/README.md new file mode 100644 index 0000000000..74198bfac5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/README.md @@ -0,0 +1,75 @@ +# Get Asset Failure — Wrong Activity for Asset Type + +This scenario reproduces a runtime `Get Credential` failure caused by +an **activity-vs-asset-type mismatch**: the workflow uses the +`Get Credential` activity (`GetRobotCredential`), which only works on +assets of type `Credential`, but the asset it targets is actually +type `Text`. Orchestrator returns HTTP **400** with the error +"Activity does not work with assets of type 'Text'. Invalid asset type." + +## What this scenario uncovers + +**Root Cause:** The `Get Credential` activity in `Main.xaml` targets +`AssetName="ApiBaseUrl"` in the `Remote Debugging` folder. The asset +exists in that folder but its `ValueType` is `Text`, not `Credential`. +The runtime mismatch is rejected before the credential read can +proceed. + +This maps to: +`references/activity-packages/system-activities/playbooks/get-asset-wrong-activity-type.md` +(the "developer selected the wrong activity when building the +workflow" branch, or the inverse "asset type was changed in +Orchestrator after the workflow was built" branch — both produce the +same error signature). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthesized UiPath project — same scaffold as `getasset-permission-denied` with `AssetName` changed to `ApiBaseUrl` (which exists but is a Text asset) | + +> **Note on fixtures.** Like sibling scenarios, fixtures here were +> authored from the documented playbook signature rather than captured +> from a real `.local/investigations/` session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression signal. + +## How this differs from sibling scenarios + +| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` (this) | +|---|---|---|---|---| +| `AssetName` | typo | correct | correct | correct | +| `FolderPath` | real folder | non-existent | real folder | real folder | +| Asset visible in folder asset list? | no (different name) | n/a | yes | **yes (but wrong type)** | +| Asset type matches activity type? | n/a | n/a | yes | **no — Text vs Credential** | +| HTTP status / error code | 404 / 1002 | 403 / 1100 | 403 / 0 | **400 / "Invalid asset type"** | +| Error message anchor | "Could not find the asset" | "Folder ... does not exist" | "You are not authorized" / "required permissions on Assets" | "does not work with assets of type" / "Invalid asset type" | +| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | `get-asset-wrong-activity-type.md` | + +This scenario forces the agent past three rule-outs (asset exists, +folder exists, permission is fine — the error isn't 403/code 0) before +landing on the type mismatch. The asset list fixture is the decisive +signal: it shows `ApiBaseUrl` is present with `ValueType: "Text"`, +which contradicts the `Get Credential` activity's expectation. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` +- Conclusion must name BOTH the activity (`Get Credential` / `GetRobotCredential`) AND the asset's actual type (`Text`) and explain that they are incompatible + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name getasset-wrong-activity-type --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/1702d2a4b1.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/1702d2a4b1.json deleted file mode 100644 index aa97629d7e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/1702d2a4b1.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "AssetList", - "Data": [ - { - "Key": "c3d4e5f6-a7b8-9012-cdef-0123456789ab", - "Name": "ApiBaseUrl", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "Base URL for the downstream API integration. Configured as a Text asset by the platform team." - }, - { - "Key": "a1b2c3d4-e5f6-7890-abcd-ef0123456789", - "Name": "myHiddenAsset", - "ValueType": "Credential", - "ValueScope": "Global", - "Description": "Credential asset used by Get Credential demos in the Remote Debugging folder." - }, - { - "Key": "b2c3d4e5-f6a7-8901-bcde-f01234567890", - "Name": "RemoteDebuggingEndpoint", - "ValueType": "Text", - "ValueScope": "Global", - "Description": "HTTPS endpoint used by remote debugging fixtures." - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/387fce62c9.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/387fce62c9.json deleted file mode 100644 index 08b4bc38a4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/387fce62c9.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6", - "StartTime": "2026-05-13T08:14:33.420Z", - "EndTime": "2026-05-13T08:14:34.512Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "4d2e7f8b-3c5a-4f9e-8d6c-1b2a3f4e5d6c", - "Info": "Get Credential: Status code: 400 (Bad Request). Orchestrator response: Activity does not work with assets of type 'Text'. Invalid asset type.\r\n\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Core.Activities.OrchestratorCommunicationException: Activity does not work with assets of type 'Text'. Invalid asset type.", - "CreationTime": "2026-05-13T08:14:33.180Z", - "ReleaseName": "AssetValueLoader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5024916, - "OrganizationUnitId": 7791621, - "FolderKey": "8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T08:14:34.512Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6", - "Id": 670953702 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/4a4cf90044.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/4a4cf90044.json deleted file mode 100644 index 48f952b0cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/4a4cf90044.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6", - "State": "Faulted", - "ProcessName": "AssetValueLoader", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T08:14:33.420Z", - "EndTime": "2026-05-13T08:14:34.512Z", - "CreationTime": "2026-05-13T08:14:33.180Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/README.md deleted file mode 100644 index 6c2ccab372..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/README.md +++ /dev/null @@ -1,74 +0,0 @@ -# Get Asset Failure — Wrong Activity for Asset Type - -This scenario reproduces a runtime `Get Credential` failure caused by -an **activity-vs-asset-type mismatch**: the workflow uses the -`Get Credential` activity (`GetRobotCredential`), which only works on -assets of type `Credential`, but the asset it targets is actually -type `Text`. Orchestrator returns HTTP **400** with the error -"Activity does not work with assets of type 'Text'. Invalid asset type." - -## What this scenario uncovers - -**Root Cause:** The `Get Credential` activity in `Main.xaml` targets -`AssetName="ApiBaseUrl"` in the `Remote Debugging` folder. The asset -exists in that folder but its `ValueType` is `Text`, not `Credential`. -The runtime mismatch is rejected before the credential read can -proceed. - -This maps to: -`references/activity-packages/system-activities/playbooks/get-asset-wrong-activity-type.md` -(the "developer selected the wrong activity when building the -workflow" branch, or the inverse "asset type was changed in -Orchestrator after the workflow was built" branch — both produce the -same error signature). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | synthesized UiPath project — same scaffold as `getasset-permission-denied` with `AssetName` changed to `ApiBaseUrl` (which exists but is a Text asset) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like sibling scenarios, fixtures here were -> authored from the documented playbook signature rather than captured -> from a real `.local/investigations/` session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression signal. - -## How this differs from sibling scenarios - -| Dimension | `name-mismatch` | `folder-scope-mismatch` | `permission-denied` | `wrong-activity-type` (this) | -|---|---|---|---|---| -| `AssetName` | typo | correct | correct | correct | -| `FolderPath` | real folder | non-existent | real folder | real folder | -| Asset visible in folder asset list? | no (different name) | n/a | yes | **yes (but wrong type)** | -| Asset type matches activity type? | n/a | n/a | yes | **no — Text vs Credential** | -| HTTP status / error code | 404 / 1002 | 403 / 1100 | 403 / 0 | **400 / "Invalid asset type"** | -| Error message anchor | "Could not find the asset" | "Folder ... does not exist" | "You are not authorized" / "required permissions on Assets" | "does not work with assets of type" / "Invalid asset type" | -| Matched playbook | `get-asset-not-found.md` | `get-asset-folder-scope-mismatch.md` | `get-asset-permission-denied.md` | `get-asset-wrong-activity-type.md` | - -This scenario forces the agent past three rule-outs (asset exists, -folder exists, permission is fine — the error isn't 403/code 0) before -landing on the type mismatch. The asset list fixture is the decisive -signal: it shows `ApiBaseUrl` is present with `ValueType: "Text"`, -which contradicts the `Get Credential` activity's expectation. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` -- Conclusion must name BOTH the activity (`Get Credential` / `GetRobotCredential`) AND the asset's actual type (`Text`) and explain that they are incompatible - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name getasset-wrong-activity-type --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/d5a6d4e894.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/d5a6d4e894.json deleted file mode 100644 index 61a1f95fa8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/d5a6d4e894.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f", - "Name": "Remote Debugging", - "Path": "Remote Debugging", - "Description": "Folder used for ad-hoc remote debugging of UiPath processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/ed86c6e056.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/ed86c6e056.json deleted file mode 100644 index 66a6e0d64e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/ed86c6e056.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T08:14:34.490Z", - "Message": "[Get Credential] Status code: 400 (Bad Request). Orchestrator response: \r\nActivity does not work with assets of type 'Text'. Invalid asset type." - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T08:14:34.505Z", - "Message": "Activity does not work with assets of type 'Text'. Invalid asset type.\r\nin Main.xaml\r\n at GetRobotCredential \"Get Credential\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/manifest.json deleted file mode 100644 index 5e5e2d5f80..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/m/r/manifest.json +++ /dev/null @@ -1,76 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the getasset-wrong-activity-type scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "d5a6d4e894.json" - }, - { - "match": "or folders list --output json", - "file": "d5a6d4e894.json" - }, - { - "match": "or jobs list --folder-key \"8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f\" --state Faulted --output json", - "file": "4a4cf90044.json" - }, - { - "match": "or jobs list --folder-key 8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f --state Faulted --output json", - "file": "4a4cf90044.json" - }, - { - "match": "or jobs list --process-name AssetValueLoader", - "file": "4a4cf90044.json" - }, - { - "match": "or jobs list --folder-path Remote Debugging", - "file": "4a4cf90044.json" - }, - { - "match": "or jobs list --folder-path \"Remote Debugging\"", - "file": "4a4cf90044.json" - }, - { - "match": "or jobs get \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\" --output json", - "file": "387fce62c9.json" - }, - { - "match": "or jobs get 1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6 --output json", - "file": "387fce62c9.json" - }, - { - "match": "or jobs logs \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\" --level Error --output json", - "file": "ed86c6e056.json" - }, - { - "match": "or jobs logs 1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6 --level Error --output json", - "file": "ed86c6e056.json" - }, - { - "match": "or assets list --folder-key 8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f --output json", - "file": "1702d2a4b1.json" - }, - { - "match": "or assets list --folder-key \"8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f\" --output json", - "file": "1702d2a4b1.json" - }, - { - "match": "or assets list --folder-path Remote Debugging", - "file": "1702d2a4b1.json" - }, - { - "match": "or assets list --folder-path \"Remote Debugging\"", - "file": "1702d2a4b1.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/uip-fixture.json new file mode 100644 index 0000000000..8eb24b759e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/data/uip-fixture.json @@ -0,0 +1,141 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetValueLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:14:33.420Z\",\r\n \"EndTime\": \"2026-05-13T08:14:34.512Z\",\r\n \"CreationTime\": \"2026-05-13T08:14:33.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetValueLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:14:33.420Z\",\r\n \"EndTime\": \"2026-05-13T08:14:34.512Z\",\r\n \"CreationTime\": \"2026-05-13T08:14:33.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T08:14:34.490Z\",\r\n \"Message\": \"[Get Credential] Status code: 400 (Bad Request). Orchestrator response: \\r\\nActivity does not work with assets of type 'Text'. Invalid asset type.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T08:14:34.505Z\",\r\n \"Message\": \"Activity does not work with assets of type 'Text'. Invalid asset type.\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote", + "Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-9012-cdef-0123456789ab\",\r\n \"Name\": \"ApiBaseUrl\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Base URL for the downstream API integration. Configured as a Text asset by the platform team.\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--process-name", + "AssetValueLoader" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetValueLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:14:33.420Z\",\r\n \"EndTime\": \"2026-05-13T08:14:34.512Z\",\r\n \"CreationTime\": \"2026-05-13T08:14:33.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AssetValueLoader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T08:14:33.420Z\",\r\n \"EndTime\": \"2026-05-13T08:14:34.512Z\",\r\n \"CreationTime\": \"2026-05-13T08:14:33.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-key", + "8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-9012-cdef-0123456789ab\",\r\n \"Name\": \"ApiBaseUrl\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Base URL for the downstream API integration. Configured as a Text asset by the platform team.\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "assets", + "list", + "--folder-path", + "Remote Debugging" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AssetList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-9012-cdef-0123456789ab\",\r\n \"Name\": \"ApiBaseUrl\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Base URL for the downstream API integration. Configured as a Text asset by the platform team.\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-abcd-ef0123456789\",\r\n \"Name\": \"myHiddenAsset\",\r\n \"ValueType\": \"Credential\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"Credential asset used by Get Credential demos in the Remote Debugging folder.\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-bcde-f01234567890\",\r\n \"Name\": \"RemoteDebuggingEndpoint\",\r\n \"ValueType\": \"Text\",\r\n \"ValueScope\": \"Global\",\r\n \"Description\": \"HTTPS endpoint used by remote debugging fixtures.\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\",\r\n \"StartTime\": \"2026-05-13T08:14:33.420Z\",\r\n \"EndTime\": \"2026-05-13T08:14:34.512Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"4d2e7f8b-3c5a-4f9e-8d6c-1b2a3f4e5d6c\",\r\n \"Info\": \"Get Credential: Status code: 400 (Bad Request). Orchestrator response: Activity does not work with assets of type 'Text'. Invalid asset type.\\r\\n\\r\\nin Main.xaml\\r\\n at GetRobotCredential \\\"Get Credential\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorCommunicationException: Activity does not work with assets of type 'Text'. Invalid asset type.\",\r\n \"CreationTime\": \"2026-05-13T08:14:33.180Z\",\r\n \"ReleaseName\": \"AssetValueLoader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5024916,\r\n \"OrganizationUnitId\": 7791621,\r\n \"FolderKey\": \"8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T08:14:34.512Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"1a2b3c4d-5e6f-7a8b-9c0d-e1f2a3b4c5d6\",\r\n \"Id\": 670953702\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8f6c5d4e-3a2b-4c9d-7e6f-1a2b3c4d5e6f\",\r\n \"Name\": \"Remote Debugging\",\r\n \"Path\": \"Remote Debugging\",\r\n \"Description\": \"Folder used for ad-hoc remote debugging of UiPath processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/task.yaml index a95ec5a25c..7226ba5f9d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getasset-wrong-activity-type/task.yaml @@ -25,22 +25,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/README.md new file mode 100644 index 0000000000..82291019c0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/README.md @@ -0,0 +1,62 @@ +# Get Queue Item Failure — Queue Not Found (No Project Source) + +This scenario replays a real diagnostic investigation: a job faulted +because its `GetQueueItem` activity targets a queue name that does not +exist in the job's folder. Orchestrator returned HTTP 404 / error code +1002, and both tenant folders hold zero queues. All identifiers +(user/workspace, queue name, folder and job keys) were neutralized before +committing; the CLI response shapes are verbatim from the recorded +session. + +## What this scenario uncovers + +**Root Cause:** The `GetQueueItem` activity "Fetch order queue item" in +`OrderQueueFetch.xaml` requests queue `OrderIntakeQueue_EU`; no queue +with that name exists anywhere in the tenant, so the lookup 404s and the +job faults. + +This maps to: +`references/activity-packages/classic-activities/playbooks/queue-operation-failed.md` +(the "queue does not exist" / "evidence cannot choose among those +branches" resolution paths) — the Add/Get Queue Item failure-family +playbook — while +`system-activities/playbooks/get-asset-not-found.md` is the near-miss the +agent must reject (same `Error code: 1002` grep signature, wrong +activity). + +## What makes it different from sys-getasset-name-mismatch + +**No `process/` dir — deliberately.** The sandbox contains no project +source, so the agent can prove the queue was absent *at run time* but +cannot prove whether the workflow's `QueueName` is a wrong hardcoded +literal, a config-fed value, or whether the intended queue was deleted. +The judge rewards keeping those claims separated (evidence-bounded +diagnosis) and penalizes asserting the unverifiable branch as fact. + +The prompt also imposes a read-only constraint; a `command_not_executed` +criterion checks no rerun/mutation happened. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +Recorded evidence path: folders list → jobs get → jobs logs → queues +list (personal) → queues list (Shared). A `jobs list` fixture is included +so exploratory listing finds the same job instead of the empty fallback. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- No job rerun or Orchestrator mutation (`command_not_executed`) +- LLM judge vs `RESOLUTION.md`: same root cause (queue absent at run + time, 404/1002) AND same fix branch (create the intended queue or + correct `QueueName` after determining intent) AND the + evidence-boundary caveat (hardcoded-vs-config / deleted-vs-never-existed + unverifiable without source) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/24539b3934.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/24539b3934.json deleted file mode 100644 index d2f9eb6420..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/24539b3934.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23", - "State": "Faulted", - "ProcessName": "OrderQueueFetch", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-01T14:31:26.380Z", - "EndTime": "2026-07-01T14:31:27.777Z", - "CreationTime": "2026-07-01T14:31:26.104Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/README.md deleted file mode 100644 index 0ae02c033c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/README.md +++ /dev/null @@ -1,61 +0,0 @@ -# Get Queue Item Failure — Queue Not Found (No Project Source) - -This scenario replays a real diagnostic investigation: a job faulted -because its `GetQueueItem` activity targets a queue name that does not -exist in the job's folder. Orchestrator returned HTTP 404 / error code -1002, and both tenant folders hold zero queues. All identifiers -(user/workspace, queue name, folder and job keys) were neutralized before -committing; the CLI response shapes are verbatim from the recorded -session. - -## What this scenario uncovers - -**Root Cause:** The `GetQueueItem` activity "Fetch order queue item" in -`OrderQueueFetch.xaml` requests queue `OrderIntakeQueue_EU`; no queue -with that name exists anywhere in the tenant, so the lookup 404s and the -job faults. - -This maps to: -`references/activity-packages/classic-activities/playbooks/queue-operation-failed.md` -(the "queue does not exist" / "evidence cannot choose among those -branches" resolution paths) — the Add/Get Queue Item failure-family -playbook — while -`system-activities/playbooks/get-asset-not-found.md` is the near-miss the -agent must reject (same `Error code: 1002` grep signature, wrong -activity). - -## What makes it different from sys-getasset-name-mismatch - -**No `process/` dir — deliberately.** The sandbox contains no project -source, so the agent can prove the queue was absent *at run time* but -cannot prove whether the workflow's `QueueName` is a wrong hardcoded -literal, a config-fed value, or whether the intended queue was deleted. -The judge rewards keeping those claims separated (evidence-bounded -diagnosis) and penalizes asserting the unverifiable branch as fact. - -The prompt also imposes a read-only constraint; a `command_not_executed` -criterion checks no rerun/mutation happened. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `data/m/r/*.json` | canned `uip` responses captured from the recorded session (identifiers neutralized) | -| `data/m/r/manifest.json` | dispatch table mapping each command pattern to its fixture | - -Recorded evidence path: folders list → jobs get → jobs logs → queues -list (personal) → queues list (Shared). A `jobs list` fixture is included -so exploratory listing finds the same job instead of the empty fallback. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- No job rerun or Orchestrator mutation (`command_not_executed`) -- LLM judge vs `RESOLUTION.md`: same root cause (queue absent at run - time, 404/1002) AND same fix branch (create the intended queue or - correct `QueueName` after determining intent) AND the - evidence-boundary caveat (hardcoded-vs-config / deleted-vs-never-existed - unverifiable without source) diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/a95a80e5c4.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/a95a80e5c4.json deleted file mode 100644 index 19d275adcc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/a95a80e5c4.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "9d4f1b2e-7a3c-4e8f-b5a6-2c1d0e9f8b7a", - "Name": "t.varga@meridianlogistics.example's workspace", - "Path": "t.varga@meridianlogistics.example's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "4a7b9c1d-2e3f-4a5b-8c7d-6e5f4a3b2c1d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/bdc1f30950.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/bdc1f30950.json deleted file mode 100644 index 034eba226b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/bdc1f30950.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "QueueList", - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/e4666ce9a5.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/e4666ce9a5.json deleted file mode 100644 index 7c782e24ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/e4666ce9a5.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-01T14:31:27.143Z", - "Message": "Fetch order queue item: Status code: 404 (Not Found). Orchestrator response: \r\nOrderIntakeQueue_EU does not exist. Error code: 1002\r\nin OrderQueueFetch.xaml\r\n at Throw \"Fetch order queue item\"\r\n at Sequence \"Sequence\"\r\n at TryCatch \"TryCatch\"\r\n at GetQueueItem \"Fetch order queue item\"\r\n at Sequence \"Fetch order queue item\"\r\n at OrderQueueFetch \"OrderQueueFetch\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.Core.Activities.OrchestratorHttpException\r\n at System.Activities.Statements.Throw.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/e6713ce2cf.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/e6713ce2cf.json deleted file mode 100644 index 21cf8d90d0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/e6713ce2cf.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23", - "StartTime": "2026-07-01T14:31:26.380Z", - "EndTime": "2026-07-01T14:31:27.777Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "6c1e8f3a-2b4d-4c5e-9a1b-3d7f0c2e4a68", - "Info": "HTTP Error\r\n\r\nCheck HTTP connectivity and verify the response error details\r\n\r\nin OrderQueueFetch.xaml\r\n at Throw \"Fetch order queue item\"\r\n at Sequence \"Sequence\"\r\n at TryCatch \"TryCatch\"\r\n at GetQueueItem \"Fetch order queue item\"\r\n at Sequence \"Fetch order queue item\"\r\n at OrderQueueFetch \"OrderQueueFetch\"\r\n\r\nUiPath.Core.Activities.OrchestratorHttpException: Status code: 404 (Not Found). Orchestrator response: \r\nOrderIntakeQueue_EU does not exist. Error code: 1002\r\n at System.Activities.Statements.Throw.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-07-01T14:31:26.104Z", - "ReleaseName": "OrderQueueFetch", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Development", - "RequiresUserInteraction": true, - "ReleaseVersionId": 5013207, - "OrganizationUnitId": 7792411, - "FolderKey": "9d4f1b2e-7a3c-4e8f-b5a6-2c1d0e9f8b7a", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "OrchestratorUserIdentity": "t.varga", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-01T14:31:27.777Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23", - "Id": 671443902 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/manifest.json deleted file mode 100644 index c6c471274d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,37 +0,0 @@ -{ - "version": 2, - "_doc": "Fixture set for a troubleshoot replay scenario; mock dispatcher data only \u2014 do not treat this file as investigation evidence.", - "rules": [ - { - "match": "or jobs get b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23", - "file": "e6713ce2cf.json" - }, - { - "match": "or jobs logs b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23", - "file": "e4666ce9a5.json" - }, - { - "match": "or jobs list", - "file": "24539b3934.json" - }, - { - "match": "or queues list", - "file": "bdc1f30950.json", - "_doc": "Both tenant folders have zero queues; one empty fixture serves every queue listing." - }, - { - "match": "or folders list", - "file": "a95a80e5c4.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..9b547fc064 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "get", + "b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23\",\r\n \"StartTime\": \"2026-07-01T14:31:26.380Z\",\r\n \"EndTime\": \"2026-07-01T14:31:27.777Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"6c1e8f3a-2b4d-4c5e-9a1b-3d7f0c2e4a68\",\r\n \"Info\": \"HTTP Error\\r\\n\\r\\nCheck HTTP connectivity and verify the response error details\\r\\n\\r\\nin OrderQueueFetch.xaml\\r\\n at Throw \\\"Fetch order queue item\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at TryCatch \\\"TryCatch\\\"\\r\\n at GetQueueItem \\\"Fetch order queue item\\\"\\r\\n at Sequence \\\"Fetch order queue item\\\"\\r\\n at OrderQueueFetch \\\"OrderQueueFetch\\\"\\r\\n\\r\\nUiPath.Core.Activities.OrchestratorHttpException: Status code: 404 (Not Found). Orchestrator response: \\r\\nOrderIntakeQueue_EU does not exist. Error code: 1002\\r\\n at System.Activities.Statements.Throw.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-07-01T14:31:26.104Z\",\r\n \"ReleaseName\": \"OrderQueueFetch\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Development\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 5013207,\r\n \"OrganizationUnitId\": 7792411,\r\n \"FolderKey\": \"9d4f1b2e-7a3c-4e8f-b5a6-2c1d0e9f8b7a\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"OrchestratorUserIdentity\": \"t.varga\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-01T14:31:27.777Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23\",\r\n \"Id\": 671443902\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-01T14:31:27.143Z\",\r\n \"Message\": \"Fetch order queue item: Status code: 404 (Not Found). Orchestrator response: \\r\\nOrderIntakeQueue_EU does not exist. Error code: 1002\\r\\nin OrderQueueFetch.xaml\\r\\n at Throw \\\"Fetch order queue item\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at TryCatch \\\"TryCatch\\\"\\r\\n at GetQueueItem \\\"Fetch order queue item\\\"\\r\\n at Sequence \\\"Fetch order queue item\\\"\\r\\n at OrderQueueFetch \\\"OrderQueueFetch\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.Core.Activities.OrchestratorHttpException\\r\\n at System.Activities.Statements.Throw.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b7e3d2a4-5f61-48c9-a0d2-4e9b1c7f8a23\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderQueueFetch\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-01T14:31:26.380Z\",\r\n \"EndTime\": \"2026-07-01T14:31:27.777Z\",\r\n \"CreationTime\": \"2026-07-01T14:31:26.104Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "queues", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueList\",\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d4f1b2e-7a3c-4e8f-b5a6-2c1d0e9f8b7a\",\r\n \"Name\": \"t.varga@meridianlogistics.example's workspace\",\r\n \"Path\": \"t.varga@meridianlogistics.example's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"4a7b9c1d-2e3f-4a5b-8c7d-6e5f4a3b2c1d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/task.yaml index 92e63c28e3..c8e8f376a7 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/sys-getqueueitem-queue-not-found/task.yaml @@ -24,22 +24,12 @@ run_limits: turn_timeout: 1800 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/README.md new file mode 100644 index 0000000000..ebdddf2ba6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/README.md @@ -0,0 +1,30 @@ +# Terminal Session — Connection Failed (unwrap AggregateException → TerminalConnectionException) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the `System.Exception` / `System.AggregateException` case of `UiPath.Terminal.Activities.TerminalSession`. + +## What this exercises + +A `TerminalSession` ("Mainframe Session") faults at connection open because the configured host is unreachable. The job ends Faulted with `System.AggregateException` whose inner exception is `UiPath.Terminal.Data.TerminalConnectionException: There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected`. The agent must **unwrap the aggregate** to the inner `TerminalConnectionException`, recognize a **connect failure** (unreachable host / connection settings) — not a workflow-logic bug — and recommend fixing host reachability / connection settings. + +Signature captured verbatim from a real Orchestrator job produced by running a `TerminalSession` against an unreachable host (`192.0.2.1`, RFC 5737 TEST-NET-1). Maps to the [terminal-session-connection-failed](../../../../../skills/uipath-troubleshoot/references/activity-packages/terminal-activities/playbooks/terminal-session-connection-failed.md) playbook. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, AggregateException + inner TerminalConnectionException) | +| `or jobs logs [--level Error]` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | empty (connect-time fault emits no spans) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence (the inner exception is in the Info / Error log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/2d930e2440.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/2d930e2440.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/2d930e2440.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/4a4735f979.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/4a4735f979.json deleted file mode 100644 index 2499a78044..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/4a4735f979.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "caf901ab-28a3-4621-bcfa-68a0447e49c8", - "StartTime": "2026-06-20T11:08:05.463Z", - "EndTime": "2026-06-20T11:08:17.150Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b4d79435-8327-4618-bc70-3db59e7f9d6f", - "Info": "One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\r\n\r\nin Main.xaml\r\n at TerminalSession \"Mainframe Session\"\r\n at Main \"Main\"\r\n\r\nSystem.AggregateException: One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\r\n ---> System.AggregateException: One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected)\r\n ---> UiPath.Terminal.Data.TerminalConnectionException: There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected\r\n at UiPath.Terminal.TerminalConnection.<>c__DisplayClass49_0.b__0(CancellationToken ct)\r\n at UiPath.Terminal.Helpers.Utils.<>c__DisplayClass3_0.b__0()\r\n at System.Threading.Tasks.Task`1.InnerInvoke()\r\n at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\r\n--- End of stack trace from previous location ---\r\n at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\r\n at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\r\n at UiPath.Terminal.Helpers.Utils.RunWithTimeoutAsync(Func`2 action, CommandOptions options)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Terminal.Activities.TerminalSession.EndExecute(NativeActivityContext context, IAsyncResult result)\r\n at UiPath.Terminal.Activities.AsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T11:08:03.143Z", - "ReleaseName": "MainframeInquiry", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1108515, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-20T11:08:17.150Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "caf901ab28a34621bcfa68a0447e49c8", - "Id": 50596528 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/56a91ad33d.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/56a91ad33d.json deleted file mode 100644 index 3ec9d5ce89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/56a91ad33d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/5be442c927.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/5be442c927.json deleted file mode 100644 index 7d21bacc91..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/5be442c927.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "caf901ab-28a3-4621-bcfa-68a0447e49c8", - "State": "Faulted", - "StartTime": "2026-06-20T11:08:05.463Z", - "EndTime": "2026-06-20T11:08:17.150Z", - "ReleaseName": "MainframeInquiry", - "Source": "Manual", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 50596528 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/9e896197b8.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/9e896197b8.json deleted file mode 100644 index 505cdcac43..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/9e896197b8.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/README.md deleted file mode 100644 index fd4108020d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Terminal Session — Connection Failed (unwrap AggregateException → TerminalConnectionException) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the `System.Exception` / `System.AggregateException` case of `UiPath.Terminal.Activities.TerminalSession`. - -## What this exercises - -A `TerminalSession` ("Mainframe Session") faults at connection open because the configured host is unreachable. The job ends Faulted with `System.AggregateException` whose inner exception is `UiPath.Terminal.Data.TerminalConnectionException: There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected`. The agent must **unwrap the aggregate** to the inner `TerminalConnectionException`, recognize a **connect failure** (unreachable host / connection settings) — not a workflow-logic bug — and recommend fixing host reachability / connection settings. - -Signature captured verbatim from a real Orchestrator job produced by running a `TerminalSession` against an unreachable host (`192.0.2.1`, RFC 5737 TEST-NET-1). Maps to the [terminal-session-connection-failed](../../../../../skills/uipath-troubleshoot/references/activity-packages/terminal-activities/playbooks/terminal-session-connection-failed.md) playbook. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, AggregateException + inner TerminalConnectionException) | -| `or jobs logs [--level Error]` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | empty (connect-time fault emits no spans) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence (the inner exception is in the Info / Error log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/e995b39aaa.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/e995b39aaa.json deleted file mode 100644 index 2a70a7ad7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/e995b39aaa.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T11:08:16.641Z", - "Message": "Mainframe Session: One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\r\nin Main.xaml\r\n at TerminalSession \"Mainframe Session\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.AggregateException\r\n at UiPath.Terminal.Activities.TerminalSession.EndExecute(NativeActivityContext context, IAsyncResult result)\r\n at UiPath.Terminal.Activities.AsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/manifest.json deleted file mode 100644 index 8b9c322bf7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of a faulted UiPath.Terminal.Activities TerminalSession job. Canned uip CLI responses captured from a real Orchestrator job; PII scrubbed. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "9e896197b8.json" - }, - { - "match": "or jobs list --folder-key 1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d --state Faulted", - "file": "5be442c927.json" - }, - { - "match": "or jobs list --folder-key 1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", - "file": "5be442c927.json" - }, - { - "match": "or jobs list", - "file": "5be442c927.json" - }, - { - "match": "or jobs get caf901ab-28a3-4621-bcfa-68a0447e49c8", - "file": "4a4735f979.json" - }, - { - "match": "or jobs logs caf901ab-28a3-4621-bcfa-68a0447e49c8 --level Error", - "file": "e995b39aaa.json" - }, - { - "match": "or jobs logs caf901ab-28a3-4621-bcfa-68a0447e49c8", - "file": "e995b39aaa.json" - }, - { - "match": "or jobs traces caf901ab-28a3-4621-bcfa-68a0447e49c8", - "file": "2d930e2440.json" - }, - { - "match": "traces spans get --job-key caf901ab-28a3-4621-bcfa-68a0447e49c8", - "file": "56a91ad33d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get caf901ab-28a3-4621-bcfa-68a0447e49c8", - "min": 1, - "description": "Agent fetches the faulted job to read its exception detail." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/uip-fixture.json new file mode 100644 index 0000000000..6a34b9e222 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"caf901ab-28a3-4621-bcfa-68a0447e49c8\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:05.463Z\",\r\n \"EndTime\": \"2026-06-20T11:08:17.150Z\",\r\n \"ReleaseName\": \"MainframeInquiry\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50596528\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "caf901ab-28a3-4621-bcfa-68a0447e49c8", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T11:08:16.641Z\",\r\n \"Message\": \"Mainframe Session: One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\\r\\nin Main.xaml\\r\\n at TerminalSession \\\"Mainframe Session\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException\\r\\n at UiPath.Terminal.Activities.TerminalSession.EndExecute(NativeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Terminal.Activities.AsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"caf901ab-28a3-4621-bcfa-68a0447e49c8\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:05.463Z\",\r\n \"EndTime\": \"2026-06-20T11:08:17.150Z\",\r\n \"ReleaseName\": \"MainframeInquiry\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50596528\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "caf901ab-28a3-4621-bcfa-68a0447e49c8" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "caf901ab-28a3-4621-bcfa-68a0447e49c8" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"caf901ab-28a3-4621-bcfa-68a0447e49c8\",\r\n \"StartTime\": \"2026-06-20T11:08:05.463Z\",\r\n \"EndTime\": \"2026-06-20T11:08:17.150Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b4d79435-8327-4618-bc70-3db59e7f9d6f\",\r\n \"Info\": \"One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\\r\\n\\r\\nin Main.xaml\\r\\n at TerminalSession \\\"Mainframe Session\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\\r\\n ---> System.AggregateException: One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected)\\r\\n ---> UiPath.Terminal.Data.TerminalConnectionException: There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected\\r\\n at UiPath.Terminal.TerminalConnection.<>c__DisplayClass49_0.b__0(CancellationToken ct)\\r\\n at UiPath.Terminal.Helpers.Utils.<>c__DisplayClass3_0.b__0()\\r\\n at System.Threading.Tasks.Task`1.InnerInvoke()\\r\\n at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)\\r\\n--- End of stack trace from previous location ---\\r\\n at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)\\r\\n at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)\\r\\n at UiPath.Terminal.Helpers.Utils.RunWithTimeoutAsync(Func`2 action, CommandOptions options)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Terminal.Activities.TerminalSession.EndExecute(NativeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Terminal.Activities.AsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T11:08:03.143Z\",\r\n \"ReleaseName\": \"MainframeInquiry\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1108515,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-20T11:08:17.150Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"caf901ab28a34621bcfa68a0447e49c8\",\r\n \"Id\": 50596528\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "caf901ab-28a3-4621-bcfa-68a0447e49c8" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T11:08:16.641Z\",\r\n \"Message\": \"Mainframe Session: One or more errors occurred. (One or more errors occurred. (There was an error connecting to terminal. Error code: UnknownError | ResultCode=UnknownError | ConnectionStatus=Disconnected))\\r\\nin Main.xaml\\r\\n at TerminalSession \\\"Mainframe Session\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException\\r\\n at UiPath.Terminal.Activities.TerminalSession.EndExecute(NativeActivityContext context, IAsyncResult result)\\r\\n at UiPath.Terminal.Activities.AsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "caf901ab-28a3-4621-bcfa-68a0447e49c8" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"caf901ab-28a3-4621-bcfa-68a0447e49c8\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:05.463Z\",\r\n \"EndTime\": \"2026-06-20T11:08:17.150Z\",\r\n \"ReleaseName\": \"MainframeInquiry\",\r\n \"Source\": \"Manual\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 50596528\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get caf901ab-28a3-4621-bcfa-68a0447e49c8", + "min": 1, + "description": "Agent fetches the faulted job to read its exception detail." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/task.yaml index 9b66af711c..7157bb1d9b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/terminal-session-connect-failed/task.yaml @@ -19,20 +19,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/README.md new file mode 100644 index 0000000000..0c6fda6b32 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/README.md @@ -0,0 +1,42 @@ +# UIA AlterIfDisabled — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the agent reached a verified resolution. The fixtures are the verbatim `uip` CLI responses captured from that session. + +## What the original session uncovered + +`NClick "Click 'Simt că am noroc(1)'"` in `ClickCase.xaml` (process `ERN`, folder `Shared`) faulted with `UiPath.UIAutomationNext.Exceptions.UiNodeDisabledElementException` ("The target element is disabled. Operation canceled.") on host `MOCK-HOST`. The selector resolved (no `NodeNotFoundException` / timeout) — the driver found the element and aborted on its own `disabled` property. + +Per the `disabled-element.md` playbook decision tree: +- Branch (A) HardwareEvents — eliminated: `InteractionMode=DebuggerApi` honors `AlterIfDisabled`. +- Branch (B) HA self-healing detected and failed to dismiss a popup — eliminated: Healing Agent was disabled on this job. +- Branch (C) HA recommendation-only inferred a popup — eliminated: Healing Agent was disabled on this job. +- Branch (D) Default — applies. + +Correct fix: set `AlterIfDisabled = True` on the failing `NClick` in `ClickCase.xaml`. The leaf element's `disabled` property is the abort cause; `InputMode` honors the property, and there is no HA popup detection that would override the default. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`ClickCase.xaml`, `Main.xaml`, etc.) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`disabled-element.md`) AND recommended `AlterIfDisabled = True` as the primary fix per branch (D) of the decision tree (with HardwareEvents and HA-popup branches eliminated) + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name uia-alter-if-disabled --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/1f2709180b.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/1f2709180b.json deleted file mode 100644 index c1578c0ac6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/1f2709180b.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 6, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/895bd23db7.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/895bd23db7.json deleted file mode 100644 index 98c29baf42..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/895bd23db7.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "49655b57-ad0a-410c-8256-38c215f9246d", - "Info": "The target element is disabled. Operation canceled.\r\n\r\nin ClickCase.xaml\r\n at NClick \"Click 'Simt c\u0103 am noroc(1)'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"ClickCase\"\r\n at ClickCase \"ClickCase\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.UiNodeDisabledElementException: The target element is disabled. Operation canceled.\r\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.GetFriendly(COMException comException)\r\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.ThrowFriendly(COMException comException)\r\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\r\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass25_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\r\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\r\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\r\n at UiPath.UIAutomationNext.Services.UiInputService.ClickAsync(IClickData data, TimeoutToken token)\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_1.<b__1>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.ClickAsync(IClickData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Extensions.IActivityActionServiceExtensions.ExecuteAsync(IActivityActionService service, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.TargetBase.ActionAsync[T](IRuntimeDataCollectionContext runtimeContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.UiPath.UIAutomationNext.Activities.IVerifyExecutionActivity.ExecuteAsync(IRuntimeContext context, IActivityData data, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyInternalAsync(IVerifyExecutionActivity activity, IRuntimeContext context, IVerifyExecutionRuntimeContext verifyContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-21T09:24:58.130Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20324882, - "EntryPointPath": "ClickCase.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T09:25:02.813Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "49655b57ad0a410c825638c215f9246d", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 64766157 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/README.md deleted file mode 100644 index af804421a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# UIA AlterIfDisabled — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the agent reached a verified resolution. The fixtures are the verbatim `uip` CLI responses captured from that session. - -## What the original session uncovered - -`NClick "Click 'Simt că am noroc(1)'"` in `ClickCase.xaml` (process `ERN`, folder `Shared`) faulted with `UiPath.UIAutomationNext.Exceptions.UiNodeDisabledElementException` ("The target element is disabled. Operation canceled.") on host `MOCK-HOST`. The selector resolved (no `NodeNotFoundException` / timeout) — the driver found the element and aborted on its own `disabled` property. - -Per the `disabled-element.md` playbook decision tree: -- Branch (A) HardwareEvents — eliminated: `InteractionMode=DebuggerApi` honors `AlterIfDisabled`. -- Branch (B) HA self-healing detected and failed to dismiss a popup — eliminated: Healing Agent was disabled on this job. -- Branch (C) HA recommendation-only inferred a popup — eliminated: Healing Agent was disabled on this job. -- Branch (D) Default — applies. - -Correct fix: set `AlterIfDisabled = True` on the failing `NClick` in `ClickCase.xaml`. The leaf element's `disabled` property is the abort cause; `InputMode` honors the property, and there is no HA popup detection that would override the default. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`ClickCase.xaml`, `Main.xaml`, etc.) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`disabled-element.md`) AND recommended `AlterIfDisabled = True` as the primary fix per branch (D) of the decision tree (with HardwareEvents and HA-popup branches eliminated) - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name uia-alter-if-disabled --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/manifest.json deleted file mode 100644 index 75230f1a23..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key 'defb8e05-e36b-4c36-bf11-0b4d08ce6cd1' --state Faulted --output json", - "file": "1f2709180b.json" - }, - { - "match": "or jobs get '49655b57-ad0a-410c-8256-38c215f9246d' --output json", - "file": "895bd23db7.json" - }, - { - "match": "or jobs get 49655b57-ad0a-410c-8256-38c215f9246d --output json", - "file": "895bd23db7.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted --output json", - "file": "1f2709180b.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/uip-fixture.json new file mode 100644 index 0000000000..0e0e94a883 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/data/uip-fixture.json @@ -0,0 +1,50 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 6,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "49655b57-ad0a-410c-8256-38c215f9246d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"Info\": \"The target element is disabled. Operation canceled.\\r\\n\\r\\nin ClickCase.xaml\\r\\n at NClick \\\"Click 'Simt c\\u0103 am noroc(1)'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"ClickCase\\\"\\r\\n at ClickCase \\\"ClickCase\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.UiNodeDisabledElementException: The target element is disabled. Operation canceled.\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.GetFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.ThrowFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass25_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.ClickAsync(IClickData data, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_1.<b__1>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.ClickAsync(IClickData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Extensions.IActivityActionServiceExtensions.ExecuteAsync(IActivityActionService service, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.TargetBase.ActionAsync[T](IRuntimeDataCollectionContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.UiPath.UIAutomationNext.Activities.IVerifyExecutionActivity.ExecuteAsync(IRuntimeContext context, IActivityData data, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyInternalAsync(IVerifyExecutionActivity activity, IRuntimeContext context, IVerifyExecutionRuntimeContext verifyContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20324882,\r\n \"EntryPointPath\": \"ClickCase.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"49655b57ad0a410c825638c215f9246d\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 64766157\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/task.yaml index da767539e5..66ffc914d9 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-alter-if-disabled/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/README.md new file mode 100644 index 0000000000..cd44ee8c5c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/README.md @@ -0,0 +1,55 @@ +# Uia Ambiguous Node Test — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent diagnosed a `NodeAmbiguousException` against a DataTables grid and +matched the `ambiguous-selector.md` playbook at high confidence. + +## What this scenario reproduces + +- Process: `ERN` (entry point `EmployeeRow.xaml`) +- Failing activity: `NClick "Click 'Airi Satou'"` inside `NApplicationCard "Edge DataTables Javascript table library"` +- Selector authored too generically (`SearchSteps='FuzzySelector'`, no `FullSelectorArgument` set): + + ``` + + + ``` + + The outer segment uniquely identifies the DataTables demo grid (`id='example'`). The inner `` matches EVERY `` cell rendered in that table — the row value `Airi Satou` appears nowhere in the selector. + +- Exception: `UiPath.UIAutomationNext.Exceptions.NodeAmbiguousException` +- Friendly message (`Strings.NodeNotFoundMultipleMatches`): "Multiple similar matches found. Could not uniquely identify the user-interface element for this action. Edit the element, run Validation, and add anchors in order to ensure the element is uniquely identified." +- Stack origin: `TargetCommonLogic.GetSearchResultAsync` → `NClick.SearchAndSetTargetAsync` → `NClick.ExecuteAsync` (no `VerifyExecutionService` frames, no Healing Agent recovery frames — the find phase short-circuits before either runs) +- Branch (B) of `ambiguous-selector.md` — repeated UI pattern without a row anchor or idx qualifier. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal UiPath project where `EmployeeRow.xaml` encodes the ambiguous-selector failure (only the failing entry point is shipped; the original project's other entry-point XAMLs and binary `.screenshots/`/`.storage/`/`.uia/` directories are stripped) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `ambiguous-selector.md` AND reached the same root cause as `RESOLUTION.md` (branch (B), row-anchor / `aaname` fix) +- Agent did NOT recommend any of the anti-patterns listed in the playbook's "Anti-patterns" section (increase Timeout, switch InteractionMode, enable Healing Agent, wrap in Retry Scope or try/catch, blind `idx='1'`) + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation /.local/investigations \ + --project C:/Users/dan.morosanu/Documents/UiPath/ERN \ + --transcript /C--work-DA-runs-uia-amb2/ \ + --scenario-name uia-ambiguous-node-test --apply +``` + +Note: re-running the generator restores the full `process/` snapshot (including the other entry-point XAMLs and binary subdirs). Strip those again after regeneration — only `EmployeeRow.xaml`, `project.json`, and `project.uiproj` are needed for the test. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/01a9c7610b.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/01a9c7610b.json deleted file mode 100644 index 253a7a9453..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/01a9c7610b.json +++ /dev/null @@ -1,152 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 10, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/0c15170a44.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/0c15170a44.json deleted file mode 100644 index 76d5fa1d16..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/0c15170a44.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "Info": "Multiple similar matches found.\r\n\r\nCould not uniquely identify the user-interface element for this action.\r\nEdit the element, run Validation, and add anchors in order to ensure the element is uniquely identified.\r\n\r\nin EmployeeRow.xaml\r\n at NClick \"Click 'Airi Satou'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge DataTables Javascript table library\"\r\n at Sequence \"Amb\"\r\n at Amb \"Amb\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeAmbiguousException: Multiple similar matches found.\r\n\r\nCould not uniquely identify the user-interface element for this action.\r\nEdit the element, run Validation, and add anchors in order to ensure the element is uniquely identified.\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-22T09:26:21.197Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20382057, - "EntryPointPath": "EmployeeRow.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-22T09:26:53.467Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "79e6d2c6ab694e32a831d80f2f6036b6", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 64888698 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/1252f7c0b7.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/1252f7c0b7.json deleted file mode 100644 index 712b1ce409..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/1252f7c0b7.json +++ /dev/null @@ -1,208 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 14, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/2069c2ec38.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/2069c2ec38.json deleted file mode 100644 index 3593762e32..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/2069c2ec38.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "e0f0dc74-f5e3-44d8-b9c8-feef5b3410fd", - "text": "UiPath Orchestrator does not provide an automatic *retry policy* feature specifically for **child jobs** out of the box. However, you can manage faulted jobs and implement retries using a combination of Orchestrator capabilities and your own logic:\n\n1. **Understand faulted jobs**\n - A job is **Faulted** if it failed to start or an unhandled error occurred during execution. Faulted jobs must be restarted manually. \n - See [Job States](https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/job-states): Faulted jobs are not auto\u2011retried by Orchestrator itself. \n\n2. **Manual / controlled retry**\n - From the **Jobs** page, you can restart any job that reached a final state (**Stopped**, **Faulted**, **Successful**) using **More Actions > Restart**. This uses the same initial settings and lets you adjust parameters before retrying. \n - This applies to both parent and child jobs if they are visible as separate jobs in Orchestrator. \n\n3. **Implementing retry logic for child jobs**\n - Build retry behavior into the **parent workflow**:\n - After starting a child job, monitor its state via activities (e.g., querying job status).\n - If the child job ends in **Faulted**, have the parent workflow:\n - Decide whether to retry (based on a counter / max attempts).\n - Start the child job again (new job) with the same or adjusted input.\n - Use Orchestrator job states (**Faulted**, **Successful**, **Stopped**) as conditions in your logic. \n\n4. **Diagnostics for failures**\n - Use **View Details** on the job to see why it faulted.\n - If **Enable Recording** was on for unattended jobs, you can [download execution media](https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/managing-jobs#downloading-execution-media) to analyze the failure before deciding on retry.\n\nBecause there is no built\u2011in child\u2011job retry policy, all retry rules (max attempts, backoff, etc.) must be implemented in your workflows or external orchestration logic.\n", - "sources": [ - "https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/job-states", - "https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs" - ], - "disclaimers": [], - "followups": [ - "How can I design a parent workflow that monitors and restarts a child job based on its Orchestrator job state?", - "Is there a recommended pattern for logging and alerting when a child job repeatedly faults in Orchestrator?", - "How do I configure and use execution recordings to troubleshoot faulted child jobs before retrying?" - ] - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/41924f0dd6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/41924f0dd6.json deleted file mode 100644 index 2e17ab7ed2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/41924f0dd6.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-22T09:26:53.374Z", - "Message": "Click 'Airi Satou': Multiple similar matches found.\r\n\r\nCould not uniquely identify the user-interface element for this action.\r\nEdit the element, run Validation, and add anchors in order to ensure the element is uniquely identified.\r\nin EmployeeRow.xaml\r\n at NClick \"Click 'Airi Satou'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge DataTables Javascript table library\"\r\n at Sequence \"Amb\"\r\n at Amb \"Amb\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.NodeAmbiguousException\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/README.md deleted file mode 100644 index 00bc7f2f18..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Uia Ambiguous Node Test — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent diagnosed a `NodeAmbiguousException` against a DataTables grid and -matched the `ambiguous-selector.md` playbook at high confidence. - -## What this scenario reproduces - -- Process: `ERN` (entry point `EmployeeRow.xaml`) -- Failing activity: `NClick "Click 'Airi Satou'"` inside `NApplicationCard "Edge DataTables Javascript table library"` -- Selector authored too generically (`SearchSteps='FuzzySelector'`, no `FullSelectorArgument` set): - - ``` - - - ``` - - The outer segment uniquely identifies the DataTables demo grid (`id='example'`). The inner `` matches EVERY `` cell rendered in that table — the row value `Airi Satou` appears nowhere in the selector. - -- Exception: `UiPath.UIAutomationNext.Exceptions.NodeAmbiguousException` -- Friendly message (`Strings.NodeNotFoundMultipleMatches`): "Multiple similar matches found. Could not uniquely identify the user-interface element for this action. Edit the element, run Validation, and add anchors in order to ensure the element is uniquely identified." -- Stack origin: `TargetCommonLogic.GetSearchResultAsync` → `NClick.SearchAndSetTargetAsync` → `NClick.ExecuteAsync` (no `VerifyExecutionService` frames, no Healing Agent recovery frames — the find phase short-circuits before either runs) -- Branch (B) of `ambiguous-selector.md` — repeated UI pattern without a row anchor or idx qualifier. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | minimal UiPath project where `EmployeeRow.xaml` encodes the ambiguous-selector failure (only the failing entry point is shipped; the original project's other entry-point XAMLs and binary `.screenshots/`/`.storage/`/`.uia/` directories are stripped) | -| `fixtures/mocks/responses/*.json` | verbatim `uip` CLI responses captured from the original session (`or folders list`, `or jobs list --state Faulted`, `or jobs get`, `or jobs logs --level Error`, `or jobs history`) | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `ambiguous-selector.md` AND reached the same root cause as `RESOLUTION.md` (branch (B), row-anchor / `aaname` fix) -- Agent did NOT recommend any of the anti-patterns listed in the playbook's "Anti-patterns" section (increase Timeout, switch InteractionMode, enable Healing Agent, wrap in Retry Scope or try/catch, blind `idx='1'`) - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation /.local/investigations \ - --project C:/Users/dan.morosanu/Documents/UiPath/ERN \ - --transcript /C--work-DA-runs-uia-amb2/ \ - --scenario-name uia-ambiguous-node-test --apply -``` - -Note: re-running the generator restores the full `process/` snapshot (including the other entry-point XAMLs and binary subdirs). Strip those again after regeneration — only `EmployeeRow.xaml`, `project.json`, and `project.uiproj` are needed for the test. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/c02dc06795.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/c02dc06795.json deleted file mode 100644 index d0ec60b1aa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/c02dc06795.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-05-22T09:26:53.587Z", - "JobId": 64888698 - }, - { - "State": "Running", - "Timestamp": "2026-05-22T09:26:21.530Z", - "JobId": 64888698 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/manifest.json deleted file mode 100644 index 94a442aaee..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "docsai ask \"how to handle child job failures in Orchestrator with retry policies for faulted jobs\" --source docs --output json", - "file": "2069c2ec38.json", - "exit_code": 1 - }, - { - "match": "or folders list --output json", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\" --state Faulted --output json", - "file": "01a9c7610b.json" - }, - { - "match": "or jobs list --folder-key \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\" --output json", - "file": "1252f7c0b7.json" - }, - { - "match": "or jobs get \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\" --output json", - "file": "0c15170a44.json" - }, - { - "match": "or jobs logs \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\" --level Error --output json", - "file": "41924f0dd6.json" - }, - { - "match": "or jobs history \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\" --output json", - "file": "c02dc06795.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/uip-fixture.json new file mode 100644 index 0000000000..3a04c6edac --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/data/uip-fixture.json @@ -0,0 +1,106 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 10,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-22T09:26:53.374Z\",\r\n \"Message\": \"Click 'Airi Satou': Multiple similar matches found.\\r\\n\\r\\nCould not uniquely identify the user-interface element for this action.\\r\\nEdit the element, run Validation, and add anchors in order to ensure the element is uniquely identified.\\r\\nin EmployeeRow.xaml\\r\\n at NClick \\\"Click 'Airi Satou'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge DataTables Javascript table library\\\"\\r\\n at Sequence \\\"Amb\\\"\\r\\n at Amb \\\"Amb\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.NodeAmbiguousException\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "docsai", + "ask", + "how to handle child job failures in Orchestrator with retry policies for faulted jobs", + "--source", + "docs", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"DocsAIAsk\",\r\n \"Data\": {\r\n \"query_id\": \"e0f0dc74-f5e3-44d8-b9c8-feef5b3410fd\",\r\n \"text\": \"UiPath Orchestrator does not provide an automatic *retry policy* feature specifically for **child jobs** out of the box. However, you can manage faulted jobs and implement retries using a combination of Orchestrator capabilities and your own logic:\\n\\n1. **Understand faulted jobs**\\n - A job is **Faulted** if it failed to start or an unhandled error occurred during execution. Faulted jobs must be restarted manually. \\n - See [Job States](https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/job-states): Faulted jobs are not auto\\u2011retried by Orchestrator itself. \\n\\n2. **Manual / controlled retry**\\n - From the **Jobs** page, you can restart any job that reached a final state (**Stopped**, **Faulted**, **Successful**) using **More Actions > Restart**. This uses the same initial settings and lets you adjust parameters before retrying. \\n - This applies to both parent and child jobs if they are visible as separate jobs in Orchestrator. \\n\\n3. **Implementing retry logic for child jobs**\\n - Build retry behavior into the **parent workflow**:\\n - After starting a child job, monitor its state via activities (e.g., querying job status).\\n - If the child job ends in **Faulted**, have the parent workflow:\\n - Decide whether to retry (based on a counter / max attempts).\\n - Start the child job again (new job) with the same or adjusted input.\\n - Use Orchestrator job states (**Faulted**, **Successful**, **Stopped**) as conditions in your logic. \\n\\n4. **Diagnostics for failures**\\n - Use **View Details** on the job to see why it faulted.\\n - If **Enable Recording** was on for unattended jobs, you can [download execution media](https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/managing-jobs#downloading-execution-media) to analyze the failure before deciding on retry.\\n\\nBecause there is no built\\u2011in child\\u2011job retry policy, all retry rules (max attempts, backoff, etc.) must be implemented in your workflows or external orchestration logic.\\n\",\r\n \"sources\": [\r\n \"https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/job-states\",\r\n \"https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs\"\r\n ],\r\n \"disclaimers\": [],\r\n \"followups\": [\r\n \"How can I design a parent workflow that monitors and restarts a child job based on its Orchestrator job state?\",\r\n \"Is there a recommended pattern for logging and alerting when a child job repeatedly faults in Orchestrator?\",\r\n \"How do I configure and use execution recordings to troubleshoot faulted child jobs before retrying?\"\r\n ]\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 14,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"Info\": \"Multiple similar matches found.\\r\\n\\r\\nCould not uniquely identify the user-interface element for this action.\\r\\nEdit the element, run Validation, and add anchors in order to ensure the element is uniquely identified.\\r\\n\\r\\nin EmployeeRow.xaml\\r\\n at NClick \\\"Click 'Airi Satou'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge DataTables Javascript table library\\\"\\r\\n at Sequence \\\"Amb\\\"\\r\\n at Amb \\\"Amb\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeAmbiguousException: Multiple similar matches found.\\r\\n\\r\\nCould not uniquely identify the user-interface element for this action.\\r\\nEdit the element, run Validation, and add anchors in order to ensure the element is uniquely identified.\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20382057,\r\n \"EntryPointPath\": \"EmployeeRow.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"79e6d2c6ab694e32a831d80f2f6036b6\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 64888698\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-22T09:26:53.587Z\",\r\n \"JobId\": 64888698\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-22T09:26:21.530Z\",\r\n \"JobId\": 64888698\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/task.yaml index e7efd5d69c..992b2b36c3 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-ambiguous-node-test/task.yaml @@ -27,22 +27,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/README.md new file mode 100644 index 0000000000..e55a6ffed4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/README.md @@ -0,0 +1,77 @@ +# UI Automation — Application Not Found (Use Application, OpenMode=Never) + +This scenario reproduces a runtime failure where a `Use Application` +(NApplicationCard) scope throws `ApplicationNotFoundException` because +its `OpenMode` is set to `Never` and the target desktop application is +not running on the robot machine. + +## What this scenario uncovers + +**Root Cause:** The `Use Application` scope ("Use AcmeBooks accounting +app") in `Main.xaml` has `OpenMode="Never"`. AcmeBooks.exe was not +running on the unattended robot when the job started (IT shut it down +for maintenance the prior evening and did not restart it). Because the +scope refuses to launch the app, it throws +`UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException` at +scope entry — before any inner `NClick` / `NTypeInto` executes. The +failure is **scope-level**, not selector-level. + +This maps to: +`references/activity-packages/ui-automation/playbooks/application-not-found.md` +(Branch A — "Workflow assumed the app was running but OpenMode=Never +blocks the launch"). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | synthetic UiPath project — a single `Main.xaml` containing the failing `Use Application` scope with `OpenMode="Never"` | + +> **Note on fixtures.** These fixtures were authored from the +> documented playbook signature — `ApplicationNotFoundException` thrown +> by `NApplicationCard` when `OpenMode=Never` and the target app is +> absent. Regenerate via `_shared/scripts/generate_scenario.py` from a +> real failed-job session before treating this test's score as a +> regression signal. + +## Why this is different from selector-failure scenarios + +| Dimension | `rpa-selector-healing-disabled` (selector) | `uia-application-not-found` (scope) | +|---|---|---| +| Exception | `SelectorNotFoundException` / `UiElementNotFoundException` | `ApplicationNotFoundException` | +| Faulted activity | element-level (`NClick`, `NTypeInto`) | scope-level (`NApplicationCard` / `Use Application`) | +| Where the failure originates | UI tree element lookup | Scope tries to attach to an application window | +| Fix domain | Selector / Healing Agent | `OpenMode` property on the scope | +| Playbook | `selector-failure-*.md` | `application-not-found.md` | + +The two failures look superficially similar (both are "UI automation +broke and the agent should consult a UI Automation playbook"), so this +scenario exercises the troubleshooting agent's ability to discriminate +between scope-level and element-level UI automation failures. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the `application-not-found` playbook AND identified + `OpenMode="Never"` in `Main.xaml` as the gating condition +- Specifically, the agent must: + - Name `ApplicationNotFoundException` (not a generic selector failure) + - Locate the `Use Application` / `NApplicationCard` scope in `Main.xaml` + - Identify `OpenMode="Never"` as the cause + - Recommend switching `OpenMode` to `IfNotOpen` (or equivalent) + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name uia-application-not-found --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/369f598163.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/369f598163.json deleted file mode 100644 index edac302cca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/369f598163.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpansGet", - "Data": [ - { - "SpanId": "01a3c5e7f9b1d2c4", - "ParentSpanId": null, - "TraceId": "2c4f8a913e7d4b6c9a1e5f8d2b7c3091", - "Name": "ExpenseImporter", - "Kind": "Internal", - "StartTime": "2026-05-12T02:00:02.220Z", - "EndTime": "2026-05-12T02:00:04.620Z", - "Status": "Error", - "Attributes": { - "workflow.entry_point": "Main.xaml", - "process.release_name": "ExpenseImporter" - } - }, - { - "SpanId": "02b4d6f8a0c2e3d5", - "ParentSpanId": "01a3c5e7f9b1d2c4", - "TraceId": "2c4f8a913e7d4b6c9a1e5f8d2b7c3091", - "Name": "Use AcmeBooks accounting app", - "Kind": "Internal", - "StartTime": "2026-05-12T02:00:02.355Z", - "EndTime": "2026-05-12T02:00:04.587Z", - "Status": "Error", - "Attributes": { - "activity.type": "UiPath.UIAutomationNext.Activities.NApplicationCard", - "activity.display_name": "Use AcmeBooks accounting app", - "activity.id_ref": "NApplicationCard_AcmeBooks", - "workflow.file": "Main.xaml", - "target_app.open_mode": "Never", - "target_app.selector": "", - "exception.type": "UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException", - "exception.message": "Could not find target application." - } - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/716cc28c12.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/716cc28c12.json deleted file mode 100644 index f549369680..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/716cc28c12.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", - "StartTime": "2026-05-12T02:00:02.140Z", - "EndTime": "2026-05-12T02:00:04.620Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "8f3c1a2d-5b7e-4f9a-b6c8-2d4e0f1a3c95", - "Info": "UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync\r\n\r\nin Main.xaml\r\n at NApplicationCard \"Use AcmeBooks accounting app\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.", - "CreationTime": "2026-05-12T02:00:01.890Z", - "ReleaseName": "ExpenseImporter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5103772, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 7791619, - "FolderKey": "f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2", - "Reference": "Nightly Expense Import", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\FINANCE-ROBOT-01", - "OrchestratorUserIdentity": "finance_robot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T02:00:04.620Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 671012845 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/8b539ae49e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/8b539ae49e.json deleted file mode 100644 index 47959dc196..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/8b539ae49e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2", - "Name": "Finance Automations", - "Path": "Finance Automations", - "Description": "Folder containing finance-team unattended jobs (expense imports, GL postings, month-end close).", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/8c45a139ad.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/8c45a139ad.json deleted file mode 100644 index e7bc3917fc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/8c45a139ad.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", - "StartTime": "2026-05-12T02:00:02.140Z", - "EndTime": "2026-05-12T02:00:04.620Z", - "State": "Faulted", - "Source": "Schedule", - "SourceType": "Schedule", - "ReleaseName": "ExpenseImporter", - "Type": "Unattended", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2", - "Info": "UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync\r\n\r\nin Main.xaml\r\n at NApplicationCard \"Use AcmeBooks accounting app\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"", - "Id": 671012845 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/975e44943a.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/975e44943a.json deleted file mode 100644 index 1f4c45aff2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/975e44943a.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T02:00:04.587Z", - "Message": "Use AcmeBooks accounting app: Could not find target application.\r\n\r\nException type: UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\r\nMessage: Could not find target application.\r\n\r\nin Main.xaml\r\n at NApplicationCard \"Use AcmeBooks accounting app\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T02:00:04.611Z", - "Message": "UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync(String appSelector, NAppOpenMode openMode, Boolean showClosestMatchesInSearchError, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync(ITargetApp targetApp, Boolean showClosestMatchesInSearchError, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(NativeActivityContext context, CancellationToken token)\r\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/README.md deleted file mode 100644 index ff18053d4e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/README.md +++ /dev/null @@ -1,76 +0,0 @@ -# UI Automation — Application Not Found (Use Application, OpenMode=Never) - -This scenario reproduces a runtime failure where a `Use Application` -(NApplicationCard) scope throws `ApplicationNotFoundException` because -its `OpenMode` is set to `Never` and the target desktop application is -not running on the robot machine. - -## What this scenario uncovers - -**Root Cause:** The `Use Application` scope ("Use AcmeBooks accounting -app") in `Main.xaml` has `OpenMode="Never"`. AcmeBooks.exe was not -running on the unattended robot when the job started (IT shut it down -for maintenance the prior evening and did not restart it). Because the -scope refuses to launch the app, it throws -`UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException` at -scope entry — before any inner `NClick` / `NTypeInto` executes. The -failure is **scope-level**, not selector-level. - -This maps to: -`references/activity-packages/ui-automation/playbooks/application-not-found.md` -(Branch A — "Workflow assumed the app was running but OpenMode=Never -blocks the launch"). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | synthetic UiPath project — a single `Main.xaml` containing the failing `Use Application` scope with `OpenMode="Never"` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** These fixtures were authored from the -> documented playbook signature — `ApplicationNotFoundException` thrown -> by `NApplicationCard` when `OpenMode=Never` and the target app is -> absent. Regenerate via `_shared/scripts/generate_scenario.py` from a -> real failed-job session before treating this test's score as a -> regression signal. - -## Why this is different from selector-failure scenarios - -| Dimension | `rpa-selector-healing-disabled` (selector) | `uia-application-not-found` (scope) | -|---|---|---| -| Exception | `SelectorNotFoundException` / `UiElementNotFoundException` | `ApplicationNotFoundException` | -| Faulted activity | element-level (`NClick`, `NTypeInto`) | scope-level (`NApplicationCard` / `Use Application`) | -| Where the failure originates | UI tree element lookup | Scope tries to attach to an application window | -| Fix domain | Selector / Healing Agent | `OpenMode` property on the scope | -| Playbook | `selector-failure-*.md` | `application-not-found.md` | - -The two failures look superficially similar (both are "UI automation -broke and the agent should consult a UI Automation playbook"), so this -scenario exercises the troubleshooting agent's ability to discriminate -between scope-level and element-level UI automation failures. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the `application-not-found` playbook AND identified - `OpenMode="Never"` in `Main.xaml` as the gating condition -- Specifically, the agent must: - - Name `ApplicationNotFoundException` (not a generic selector failure) - - Locate the `Use Application` / `NApplicationCard` scope in `Main.xaml` - - Identify `OpenMode="Never"` as the cause - - Recommend switching `OpenMode` to `IfNotOpen` (or equivalent) - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name uia-application-not-found --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/a086aaa016.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/a086aaa016.json deleted file mode 100644 index 58aae342d7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/a086aaa016.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-12T02:00:02.220Z", - "Message": "ExpenseImporter started - importing yesterday's expense reports into AcmeBooks" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-12T02:00:02.301Z", - "Message": "Execution started: Main.xaml" - }, - { - "Level": "Trace", - "TimeStamp": "2026-05-12T02:00:02.355Z", - "Message": "Activity start: 'Use AcmeBooks accounting app' (NApplicationCard) — OpenMode=Never, Selector=" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T02:00:04.587Z", - "Message": "Use AcmeBooks accounting app: Could not find target application.\r\n\r\nException type: UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\r\nMessage: Could not find target application.\r\n\r\nin Main.xaml\r\n at NApplicationCard \"Use AcmeBooks accounting app\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T02:00:04.611Z", - "Message": "UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync(String appSelector, NAppOpenMode openMode, Boolean showClosestMatchesInSearchError, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync(ITargetApp targetApp, Boolean showClosestMatchesInSearchError, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(NativeActivityContext context, CancellationToken token)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/ffabb76cf2.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/ffabb76cf2.json deleted file mode 100644 index 7303d75f5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/ffabb76cf2.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Pending", - "TimeStamp": "2026-05-12T02:00:01.890Z", - "PrincipalName": "TriggerService", - "Reason": "Triggered by schedule 'Nightly Expense Import'" - }, - { - "State": "Running", - "TimeStamp": "2026-05-12T02:00:02.140Z", - "PrincipalName": "UIPATH\\FINANCE-ROBOT-01", - "Reason": "Picked up by robot" - }, - { - "State": "Faulted", - "TimeStamp": "2026-05-12T02:00:04.620Z", - "PrincipalName": "UIPATH\\FINANCE-ROBOT-01", - "Reason": "UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/manifest.json deleted file mode 100644 index cd4de96d38..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,72 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixtures hand-authored from the application-not-found playbook signature (ApplicationNotFoundException thrown by NApplicationCard when OpenMode=Never). Replace with verbatim captures from a real .local/investigations/ before treating as a regression signal (see ../_shared/scripts/generate_scenario.py).", - "rules": [ - { - "match": "or folders list --output json", - "file": "8b539ae49e.json" - }, - { - "match": "or jobs list --folder-key \"f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2\" --state Faulted --output json", - "file": "8c45a139ad.json" - }, - { - "match": "or jobs list --folder-key f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2 --state Faulted --output json", - "file": "8c45a139ad.json" - }, - { - "match": "or jobs get \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\" --output json", - "file": "716cc28c12.json" - }, - { - "match": "or jobs get 2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091 --output json", - "file": "716cc28c12.json" - }, - { - "match": "or jobs logs \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\" --level Error --output json", - "file": "975e44943a.json" - }, - { - "match": "or jobs logs 2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091 --level Error --output json", - "file": "975e44943a.json" - }, - { - "match": "or jobs logs \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\" --output json", - "file": "a086aaa016.json" - }, - { - "match": "or jobs logs 2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091 --output json", - "file": "a086aaa016.json" - }, - { - "match": "or jobs history \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\" --output json", - "file": "ffabb76cf2.json" - }, - { - "match": "or jobs history 2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091 --output json", - "file": "ffabb76cf2.json" - }, - { - "match": "traces spans get --job-key 2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091 --output json", - "file": "369f598163.json" - }, - { - "match": "traces spans get 2c4f8a913e7d4b6c9a1e5f8d2b7c3091 --output json", - "file": "369f598163.json" - }, - { - "match": "traces spans get 2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091 --output json", - "file": "369f598163.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns []. Lets the agent explore beyond the recorded path." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..8373a34661 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/data/uip-fixture.json @@ -0,0 +1,131 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\",\r\n \"StartTime\": \"2026-05-12T02:00:02.140Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.620Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"ReleaseName\": \"ExpenseImporter\",\r\n \"Type\": \"Unattended\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2\",\r\n \"Info\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync\\r\\n\\r\\nin Main.xaml\\r\\n at NApplicationCard \\\"Use AcmeBooks accounting app\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\",\r\n \"Id\": 671012845\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T02:00:04.587Z\",\r\n \"Message\": \"Use AcmeBooks accounting app: Could not find target application.\\r\\n\\r\\nException type: UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\\r\\nMessage: Could not find target application.\\r\\n\\r\\nin Main.xaml\\r\\n at NApplicationCard \\\"Use AcmeBooks accounting app\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T02:00:04.611Z\",\r\n \"Message\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync(String appSelector, NAppOpenMode openMode, Boolean showClosestMatchesInSearchError, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync(ITargetApp targetApp, Boolean showClosestMatchesInSearchError, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(NativeActivityContext context, CancellationToken token)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpansGet\",\r\n \"Data\": [\r\n {\r\n \"SpanId\": \"01a3c5e7f9b1d2c4\",\r\n \"ParentSpanId\": null,\r\n \"TraceId\": \"2c4f8a913e7d4b6c9a1e5f8d2b7c3091\",\r\n \"Name\": \"ExpenseImporter\",\r\n \"Kind\": \"Internal\",\r\n \"StartTime\": \"2026-05-12T02:00:02.220Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.620Z\",\r\n \"Status\": \"Error\",\r\n \"Attributes\": {\r\n \"workflow.entry_point\": \"Main.xaml\",\r\n \"process.release_name\": \"ExpenseImporter\"\r\n }\r\n },\r\n {\r\n \"SpanId\": \"02b4d6f8a0c2e3d5\",\r\n \"ParentSpanId\": \"01a3c5e7f9b1d2c4\",\r\n \"TraceId\": \"2c4f8a913e7d4b6c9a1e5f8d2b7c3091\",\r\n \"Name\": \"Use AcmeBooks accounting app\",\r\n \"Kind\": \"Internal\",\r\n \"StartTime\": \"2026-05-12T02:00:02.355Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.587Z\",\r\n \"Status\": \"Error\",\r\n \"Attributes\": {\r\n \"activity.type\": \"UiPath.UIAutomationNext.Activities.NApplicationCard\",\r\n \"activity.display_name\": \"Use AcmeBooks accounting app\",\r\n \"activity.id_ref\": \"NApplicationCard_AcmeBooks\",\r\n \"workflow.file\": \"Main.xaml\",\r\n \"target_app.open_mode\": \"Never\",\r\n \"target_app.selector\": \"\",\r\n \"exception.type\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\",\r\n \"exception.message\": \"Could not find target application.\"\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\",\r\n \"StartTime\": \"2026-05-12T02:00:02.140Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.620Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"8f3c1a2d-5b7e-4f9a-b6c8-2d4e0f1a3c95\",\r\n \"Info\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync\\r\\n\\r\\nin Main.xaml\\r\\n at NApplicationCard \\\"Use AcmeBooks accounting app\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\",\r\n \"CreationTime\": \"2026-05-12T02:00:01.890Z\",\r\n \"ReleaseName\": \"ExpenseImporter\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5103772,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 7791619,\r\n \"FolderKey\": \"f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2\",\r\n \"Reference\": \"Nightly Expense Import\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\FINANCE-ROBOT-01\",\r\n \"OrchestratorUserIdentity\": \"finance_robot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-12T02:00:04.620Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 671012845\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T02:00:02.220Z\",\r\n \"Message\": \"ExpenseImporter started - importing yesterday's expense reports into AcmeBooks\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-12T02:00:02.301Z\",\r\n \"Message\": \"Execution started: Main.xaml\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-05-12T02:00:02.355Z\",\r\n \"Message\": \"Activity start: 'Use AcmeBooks accounting app' (NApplicationCard) — OpenMode=Never, Selector=\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T02:00:04.587Z\",\r\n \"Message\": \"Use AcmeBooks accounting app: Could not find target application.\\r\\n\\r\\nException type: UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\\r\\nMessage: Could not find target application.\\r\\n\\r\\nin Main.xaml\\r\\n at NApplicationCard \\\"Use AcmeBooks accounting app\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T02:00:04.611Z\",\r\n \"Message\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetAppNotFoundExceptionAsync(String appSelector, NAppOpenMode openMode, Boolean showClosestMatchesInSearchError, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ProcessAppNotFoundAsync(ITargetApp targetApp, Boolean showClosestMatchesInSearchError, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(NativeActivityContext context, CancellationToken token)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Pending\",\r\n \"TimeStamp\": \"2026-05-12T02:00:01.890Z\",\r\n \"PrincipalName\": \"TriggerService\",\r\n \"Reason\": \"Triggered by schedule 'Nightly Expense Import'\"\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"TimeStamp\": \"2026-05-12T02:00:02.140Z\",\r\n \"PrincipalName\": \"UIPATH\\\\FINANCE-ROBOT-01\",\r\n \"Reason\": \"Picked up by robot\"\r\n },\r\n {\r\n \"State\": \"Faulted\",\r\n \"TimeStamp\": \"2026-05-12T02:00:04.620Z\",\r\n \"PrincipalName\": \"UIPATH\\\\FINANCE-ROBOT-01\",\r\n \"Reason\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException: Could not find target application.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "2c4f8a913e7d4b6c9a1e5f8d2b7c3091", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpansGet\",\r\n \"Data\": [\r\n {\r\n \"SpanId\": \"01a3c5e7f9b1d2c4\",\r\n \"ParentSpanId\": null,\r\n \"TraceId\": \"2c4f8a913e7d4b6c9a1e5f8d2b7c3091\",\r\n \"Name\": \"ExpenseImporter\",\r\n \"Kind\": \"Internal\",\r\n \"StartTime\": \"2026-05-12T02:00:02.220Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.620Z\",\r\n \"Status\": \"Error\",\r\n \"Attributes\": {\r\n \"workflow.entry_point\": \"Main.xaml\",\r\n \"process.release_name\": \"ExpenseImporter\"\r\n }\r\n },\r\n {\r\n \"SpanId\": \"02b4d6f8a0c2e3d5\",\r\n \"ParentSpanId\": \"01a3c5e7f9b1d2c4\",\r\n \"TraceId\": \"2c4f8a913e7d4b6c9a1e5f8d2b7c3091\",\r\n \"Name\": \"Use AcmeBooks accounting app\",\r\n \"Kind\": \"Internal\",\r\n \"StartTime\": \"2026-05-12T02:00:02.355Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.587Z\",\r\n \"Status\": \"Error\",\r\n \"Attributes\": {\r\n \"activity.type\": \"UiPath.UIAutomationNext.Activities.NApplicationCard\",\r\n \"activity.display_name\": \"Use AcmeBooks accounting app\",\r\n \"activity.id_ref\": \"NApplicationCard_AcmeBooks\",\r\n \"workflow.file\": \"Main.xaml\",\r\n \"target_app.open_mode\": \"Never\",\r\n \"target_app.selector\": \"\",\r\n \"exception.type\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\",\r\n \"exception.message\": \"Could not find target application.\"\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "2c4f8a91-3e7d-4b6c-9a1e-5f8d2b7c3091", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpansGet\",\r\n \"Data\": [\r\n {\r\n \"SpanId\": \"01a3c5e7f9b1d2c4\",\r\n \"ParentSpanId\": null,\r\n \"TraceId\": \"2c4f8a913e7d4b6c9a1e5f8d2b7c3091\",\r\n \"Name\": \"ExpenseImporter\",\r\n \"Kind\": \"Internal\",\r\n \"StartTime\": \"2026-05-12T02:00:02.220Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.620Z\",\r\n \"Status\": \"Error\",\r\n \"Attributes\": {\r\n \"workflow.entry_point\": \"Main.xaml\",\r\n \"process.release_name\": \"ExpenseImporter\"\r\n }\r\n },\r\n {\r\n \"SpanId\": \"02b4d6f8a0c2e3d5\",\r\n \"ParentSpanId\": \"01a3c5e7f9b1d2c4\",\r\n \"TraceId\": \"2c4f8a913e7d4b6c9a1e5f8d2b7c3091\",\r\n \"Name\": \"Use AcmeBooks accounting app\",\r\n \"Kind\": \"Internal\",\r\n \"StartTime\": \"2026-05-12T02:00:02.355Z\",\r\n \"EndTime\": \"2026-05-12T02:00:04.587Z\",\r\n \"Status\": \"Error\",\r\n \"Attributes\": {\r\n \"activity.type\": \"UiPath.UIAutomationNext.Activities.NApplicationCard\",\r\n \"activity.display_name\": \"Use AcmeBooks accounting app\",\r\n \"activity.id_ref\": \"NApplicationCard_AcmeBooks\",\r\n \"workflow.file\": \"Main.xaml\",\r\n \"target_app.open_mode\": \"Never\",\r\n \"target_app.selector\": \"\",\r\n \"exception.type\": \"UiPath.UIAutomationNext.Exceptions.ApplicationNotFoundException\",\r\n \"exception.message\": \"Could not find target application.\"\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f3e8d4b1-9c2a-4f5e-b8d7-c1a0e6f9b3d2\",\r\n \"Name\": \"Finance Automations\",\r\n \"Path\": \"Finance Automations\",\r\n \"Description\": \"Folder containing finance-team unattended jobs (expense imports, GL postings, month-end close).\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b8e2c145-7d3a-49b6-a2c8-1f4b5e8a9c01\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/task.yaml index 9aae8ff276..94cd2fec8f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-not-found/task.yaml @@ -23,21 +23,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/README.md new file mode 100644 index 0000000000..2782cc17c5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/README.md @@ -0,0 +1,36 @@ +# Uia Application Open Failed — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The "GitHub Desktop" Use Application/Browser scope in `DesktopApp.xaml` has a hard-coded `File path` pointing at a developer-machine install location — `C:\Users\original_user\AppData\Local\GitHubDesktop\app-3.5.12\githubdesktop.exe` — that does not exist on the robot machine MOCK-HOST. Because the scope is set to open the app (no `Open` mode set, so it defaults to "If not open"), it tried to launch that exact executable, the file was not there, and the scope threw `ApplicationOpenException` before any activity inside it could run. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name uia-application-open-failed --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/017b497036.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/017b497036.json deleted file mode 100644 index 426d03b638..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/017b497036.json +++ /dev/null @@ -1,421 +0,0 @@ -EXIT=0 ----ERR--- ----STDOUT--- -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e4237b15-1574-4c7a-8657-fe0b0be2aa75", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-05T07:34:54.547Z", - "EndTime": "2026-06-05T07:34:56.163Z", - "CreationTime": "2026-06-05T07:34:54.547Z" - }, - { - "Key": "8e2c5979-e05f-4e03-b6d1-cda02aa8450c", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T15:10:44.610Z", - "EndTime": "2026-06-03T15:11:17.967Z", - "CreationTime": "2026-06-03T15:10:44.610Z" - }, - { - "Key": "cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T11:56:44.780Z", - "EndTime": "2026-06-03T11:56:52.850Z", - "CreationTime": "2026-06-03T11:56:44.780Z" - }, - { - "Key": "025d0847-ce15-494f-91d6-04be2a19be6d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:28:08.133Z", - "EndTime": "2026-06-03T09:29:10.690Z", - "CreationTime": "2026-06-03T09:28:08.133Z" - }, - { - "Key": "90058dec-f287-4b4d-8272-1b7cf72e0c0d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-03T09:17:53.650Z", - "EndTime": "2026-06-03T09:18:57.633Z", - "CreationTime": "2026-06-03T09:17:53.650Z" - }, - { - "Key": "b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d", - "State": "Pending", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-29T12:28:56.660Z" - }, - { - "Key": "ccc3e841-bdcf-4501-9d30-f112f098f630", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T13:09:18.353Z", - "EndTime": "2026-05-28T13:10:10.310Z", - "CreationTime": "2026-05-28T12:48:07.547Z" - }, - { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T08:00:50.870Z", - "EndTime": "2026-05-28T08:01:02.417Z", - "CreationTime": "2026-05-27T16:24:24.603Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-26T08:48:20.617Z", - "EndTime": "2026-05-26T08:48:20.617Z", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 29, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/14d9b4ac94.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/14d9b4ac94.json deleted file mode 100644 index 66411b8262..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/14d9b4ac94.json +++ /dev/null @@ -1,48 +0,0 @@ -GET_EXIT=0 -===DETAILS=== -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e4237b15-1574-4c7a-8657-fe0b0be2aa75", - "StartTime": "2026-06-05T07:34:54.547Z", - "EndTime": "2026-06-05T07:34:56.163Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "e4237b15-1574-4c7a-8657-fe0b0be2aa75", - "Info": "Could not open target application.\r\nSpecified file path 'C:\\Users\\original_user\\AppData\\Local\\GitHubDesktop\\app-3.5.12\\githubdesktop.exe' does not refer to an existing file.\r\n\r\nin DesktopApp.xaml\r\n at NApplicationCard \"GitHub Desktop\"\r\n at Sequence \"DesktopApp\"\r\n at DesktopApp \"DesktopApp\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.ApplicationOpenException: Could not open target application.\r\nSpecified file path 'C:\\Users\\original_user\\AppData\\Local\\GitHubDesktop\\app-3.5.12\\githubdesktop.exe' does not refer to an existing file.\r\n at UiPath.UIAutomationNext.Services.DriverApplicationService.IsOpenApplicationEnabled(ITargetApp targetApplication)\r\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationInternalAsync(ITargetApp target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationAsync(ITargetApp target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetMainWindowByTargetApp(IActivityContextWrapper context, ITargetApp targetApp, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetRuntimeData(IActivityContextWrapper context, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OpenOrAttach(IActivityContextWrapper context, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-05T07:34:54.547Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20846849, - "EntryPointPath": "DesktopApp.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\ORIGINAL_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-05T07:34:56.163Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e4237b1515744c7a8657fe0b0be2aa75", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 66152281 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/2748e8d2cc.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/2748e8d2cc.json deleted file mode 100644 index de9b6ff192..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/2748e8d2cc.json +++ /dev/null @@ -1,31 +0,0 @@ -EXIT=0 ----ERR--- ----STDOUT HEAD--- -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/README.md deleted file mode 100644 index 21b7b3119e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Uia Application Open Failed — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The "GitHub Desktop" Use Application/Browser scope in `DesktopApp.xaml` has a hard-coded `File path` pointing at a developer-machine install location — `C:\Users\original_user\AppData\Local\GitHubDesktop\app-3.5.12\githubdesktop.exe` — that does not exist on the robot machine MOCK-HOST. Because the scope is set to open the app (no `Open` mode set, so it defaults to "If not open"), it tried to launch that exact executable, the file was not there, and the scope threw `ApplicationOpenException` before any activity inside it could run. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name uia-application-open-failed --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/a86ba118d0.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/a86ba118d0.json deleted file mode 100644 index b030c0b056..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/a86ba118d0.json +++ /dev/null @@ -1,18 +0,0 @@ -HIST_EXIT=0 -===HISTORY=== -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-05T07:34:56.183Z", - "JobId": 66152281 - }, - { - "State": "Running", - "Timestamp": "2026-06-05T07:34:54.750Z", - "JobId": 66152281 - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/b203483fc2.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/b203483fc2.json deleted file mode 100644 index 48ed9419d8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/b203483fc2.json +++ /dev/null @@ -1,16 +0,0 @@ -TRACES_EXIT=0 ----ERR--- ----HEAD--- -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-7169-966774d99b29", - "TraceId": "e4237b15-1574-4c7a-8657-fe0b0be2aa75", - "ParentId": null, - "Name": "Robot Job e4237b15-1574-4c7a-8657-fe0b0be2aa75", - "StartTime": "2026-06-05T07:34:55.2903136Z", - "EndTime": "2026-06-05T07:34:56.0128536Z", - "Attributes": "{\"Exception\":\"UiPath.UIAutomationNext.Exceptions.ApplicationOpenException: Could not open target application.\\r\\nSpecified file path 'C:\\\\Users\\\\original_user\\\\AppData\\\\Local\\\\GitHubDesktop\\\\app-3.5.12\\\\githubdesktop.exe' does not refer to an existing file.\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.IsOpenApplicationEnabled(ITargetApp targetApplication)\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationInternalAsync(ITargetApp target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationAsync(ITargetApp target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetMainWindowByTargetApp(IActivityContextWrapper context, ITargetApp targetApp, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetRuntimeData(IActivityContextWrapper context, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OpenOrAttach(IActivityContextWrapper context, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper context \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/d69f6d8f86.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/d69f6d8f86.json deleted file mode 100644 index 4620f84c5e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/d69f6d8f86.json +++ /dev/null @@ -1,13 +0,0 @@ -LOGS_EXIT=0 -===LOGS=== -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-05T07:34:56.003Z", - "Message": "GitHub Desktop: Could not open target application.\r\nSpecified file path 'C:\\Users\\original_user\\AppData\\Local\\GitHubDesktop\\app-3.5.12\\githubdesktop.exe' does not refer to an existing file.\r\nin DesktopApp.xaml\r\n at NApplicationCard \"GitHub Desktop\"\r\n at Sequence \"DesktopApp\"\r\n at DesktopApp \"DesktopApp\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.ApplicationOpenException\r\n at UiPath.UIAutomationNext.Services.DriverApplicationService.IsOpenApplicationEnabled(ITargetApp targetApplication)\r\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationInternalAsync(ITargetApp target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationAsync(ITargetApp target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetMainWindowByTargetApp(IActivityContextWrapper context, ITargetApp targetApp, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetRuntimeData(IActivityContextWrapper context, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OpenOrAttach(IActivityContextWrapper context, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/manifest.json deleted file mode 100644 index 773f96b808..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/m/r/manifest.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "2748e8d2cc.json" - }, - { - "match": "or jobs list --folder-key \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\" --process-name \"ERN\" --output json", - "file": "017b497036.json" - }, - { - "match": "or jobs get \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\" --output json", - "file": "14d9b4ac94.json" - }, - { - "match": "or jobs logs \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\" --level Error --output json", - "file": "d69f6d8f86.json" - }, - { - "match": "or jobs history \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\" --output json", - "file": "a86ba118d0.json" - }, - { - "match": "or jobs traces \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\" --output json", - "file": "b203483fc2.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/uip-fixture.json new file mode 100644 index 0000000000..625db7303a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/data/uip-fixture.json @@ -0,0 +1,91 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--process-name", + "ERN", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n---ERR---\r\n---STDOUT---\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-05T07:34:54.547Z\",\r\n \"EndTime\": \"2026-06-05T07:34:56.163Z\",\r\n \"CreationTime\": \"2026-06-05T07:34:54.547Z\"\r\n },\r\n {\r\n \"Key\": \"8e2c5979-e05f-4e03-b6d1-cda02aa8450c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T15:10:44.610Z\",\r\n \"EndTime\": \"2026-06-03T15:11:17.967Z\",\r\n \"CreationTime\": \"2026-06-03T15:10:44.610Z\"\r\n },\r\n {\r\n \"Key\": \"cb68d1ab-5a5b-4e5e-9416-2c571e29c5aa\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T11:56:44.780Z\",\r\n \"EndTime\": \"2026-06-03T11:56:52.850Z\",\r\n \"CreationTime\": \"2026-06-03T11:56:44.780Z\"\r\n },\r\n {\r\n \"Key\": \"025d0847-ce15-494f-91d6-04be2a19be6d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:28:08.133Z\",\r\n \"EndTime\": \"2026-06-03T09:29:10.690Z\",\r\n \"CreationTime\": \"2026-06-03T09:28:08.133Z\"\r\n },\r\n {\r\n \"Key\": \"90058dec-f287-4b4d-8272-1b7cf72e0c0d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-03T09:17:53.650Z\",\r\n \"EndTime\": \"2026-06-03T09:18:57.633Z\",\r\n \"CreationTime\": \"2026-06-03T09:17:53.650Z\"\r\n },\r\n {\r\n \"Key\": \"b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-29T12:28:56.660Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T08:00:50.870Z\",\r\n \"EndTime\": \"2026-05-28T08:01:02.417Z\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 29,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e4237b15-1574-4c7a-8657-fe0b0be2aa75", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "LOGS_EXIT=0\r\n===LOGS===\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-05T07:34:56.003Z\",\r\n \"Message\": \"GitHub Desktop: Could not open target application.\\r\\nSpecified file path 'C:\\\\Users\\\\original_user\\\\AppData\\\\Local\\\\GitHubDesktop\\\\app-3.5.12\\\\githubdesktop.exe' does not refer to an existing file.\\r\\nin DesktopApp.xaml\\r\\n at NApplicationCard \\\"GitHub Desktop\\\"\\r\\n at Sequence \\\"DesktopApp\\\"\\r\\n at DesktopApp \\\"DesktopApp\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.ApplicationOpenException\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.IsOpenApplicationEnabled(ITargetApp targetApplication)\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationInternalAsync(ITargetApp target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationAsync(ITargetApp target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetMainWindowByTargetApp(IActivityContextWrapper context, ITargetApp targetApp, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetRuntimeData(IActivityContextWrapper context, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OpenOrAttach(IActivityContextWrapper context, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e4237b15-1574-4c7a-8657-fe0b0be2aa75", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "GET_EXIT=0\r\n===DETAILS===\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"StartTime\": \"2026-06-05T07:34:54.547Z\",\r\n \"EndTime\": \"2026-06-05T07:34:56.163Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"Info\": \"Could not open target application.\\r\\nSpecified file path 'C:\\\\Users\\\\original_user\\\\AppData\\\\Local\\\\GitHubDesktop\\\\app-3.5.12\\\\githubdesktop.exe' does not refer to an existing file.\\r\\n\\r\\nin DesktopApp.xaml\\r\\n at NApplicationCard \\\"GitHub Desktop\\\"\\r\\n at Sequence \\\"DesktopApp\\\"\\r\\n at DesktopApp \\\"DesktopApp\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.ApplicationOpenException: Could not open target application.\\r\\nSpecified file path 'C:\\\\Users\\\\original_user\\\\AppData\\\\Local\\\\GitHubDesktop\\\\app-3.5.12\\\\githubdesktop.exe' does not refer to an existing file.\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.IsOpenApplicationEnabled(ITargetApp targetApplication)\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationInternalAsync(ITargetApp target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationAsync(ITargetApp target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetMainWindowByTargetApp(IActivityContextWrapper context, ITargetApp targetApp, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetRuntimeData(IActivityContextWrapper context, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OpenOrAttach(IActivityContextWrapper context, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-05T07:34:54.547Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20846849,\r\n \"EntryPointPath\": \"DesktopApp.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\ORIGINAL_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-05T07:34:56.163Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e4237b1515744c7a8657fe0b0be2aa75\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 66152281\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "history", + "e4237b15-1574-4c7a-8657-fe0b0be2aa75", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "HIST_EXIT=0\r\n===HISTORY===\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-05T07:34:56.183Z\",\r\n \"JobId\": 66152281\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-05T07:34:54.750Z\",\r\n \"JobId\": 66152281\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "e4237b15-1574-4c7a-8657-fe0b0be2aa75", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "TRACES_EXIT=0\r\n---ERR---\r\n---HEAD---\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-7169-966774d99b29\",\r\n \"TraceId\": \"e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job e4237b15-1574-4c7a-8657-fe0b0be2aa75\",\r\n \"StartTime\": \"2026-06-05T07:34:55.2903136Z\",\r\n \"EndTime\": \"2026-06-05T07:34:56.0128536Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.UIAutomationNext.Exceptions.ApplicationOpenException: Could not open target application.\\\\r\\\\nSpecified file path 'C:\\\\\\\\Users\\\\\\\\original_user\\\\\\\\AppData\\\\\\\\Local\\\\\\\\GitHubDesktop\\\\\\\\app-3.5.12\\\\\\\\githubdesktop.exe' does not refer to an existing file.\\\\r\\\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.IsOpenApplicationEnabled(ITargetApp targetApplication)\\\\r\\\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationInternalAsync(ITargetApp target, CancellationToken token)\\\\r\\\\n at UiPath.UIAutomationNext.Services.DriverApplicationService.ActivateApplicationAsync(ITargetApp target, CancellationToken token)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetMainWindowByTargetApp(IActivityContextWrapper context, ITargetApp targetApp, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.GetRuntimeData(IActivityContextWrapper context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OpenOrAttach(IActivityContextWrapper context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper context" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n---ERR---\r\n---STDOUT HEAD---\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/task.yaml index 2ba0d8aafe..096d754331 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-application-open-failed/task.yaml @@ -18,22 +18,12 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/README.md new file mode 100644 index 0000000000..d36896af1b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/README.md @@ -0,0 +1,34 @@ +# UIAutomationNext Use Browser — Cannot Communicate With the Browser (Runtime) + +Runtime troubleshooting scenario for `UiPath.UIAutomationNext.Activities.NApplicationCard` (the modern +Use Application/Browser container) in a **browser** configuration. + +## What this scenario exercises + +An unattended job faults on the `Use Browser` scope with `Cannot communicate with the browser.` The +process ran clean on the same schedule and machine for weeks and the automation was not changed. The +agent must recognize this as a **broken browser-automation channel** (the browser is up, but the UiPath +extension / native-messaging bridge cannot carry commands) rather than a launch failure +(`ApplicationOpenException`) or a not-found (`ApplicationNotFoundException`), and prescribe the +extension / native-messaging / browser↔package-version remediation. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` with a `Use Browser` `NApplicationCard` (Chrome) wrapping `NTypeInto` / `NClick`; current, aligned package pins (no design-time skew) | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the broken browser-automation channel and the remediation branches (UiPath extension + enabled/allowed-in-incognito, native-messaging host not blocked by antivirus/policy, browser↔package + version alignment), and ruled out launch-failure / not-found. + +Playbook: `references/activity-packages/ui-automation/playbooks/browser-communication-failed.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/244ab94cfe.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/244ab94cfe.json deleted file mode 100644 index 6caf2beb6f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/244ab94cfe.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-19T02:15:41.870Z", - "JobId": 66540211 - }, - { - "State": "Running", - "Timestamp": "2026-07-19T02:15:04.640Z", - "JobId": 66540211 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/3672254a58.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/3672254a58.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/3672254a58.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/464db476bb.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/464db476bb.json deleted file mode 100644 index e97493ad16..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/464db476bb.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7d2e5a10-9c34-4f62-b18a-3e6f0c9d2b47", - "StartTime": "2026-07-19T02:15:04.110Z", - "EndTime": "2026-07-19T02:15:41.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Use CustomerHub portal: Cannot communicate with the browser.\r\n\r\nin Main.xaml\r\n at NApplicationCard \"Use CustomerHub portal\"\r\n at Sequence \"Main Sequence\"\r\n at DynamicActivity \"Main\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.UiAutomationExecutionException: Cannot communicate with the browser.\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.d__.MoveNext()", - "CreationTime": "2026-07-19T02:15:04.110Z", - "ReleaseName": "CustomerHubSync", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-19T02:15:41.870Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7d2e5a109c344f62b18a3e6f0c9d2b47", - "Id": 66540211 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/6059da6ae1.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/6059da6ae1.json deleted file mode 100644 index 9be9b6acf7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/6059da6ae1.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7d2e5a10-9c34-4f62-b18a-3e6f0c9d2b47", - "State": "Faulted", - "ProcessName": "CustomerHubSync", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-19T02:15:04.110Z", - "EndTime": "2026-07-19T02:15:41.870Z", - "CreationTime": "2026-07-19T02:15:04.110Z" - }, - { - "Key": "5a9c1e83-2d47-4b16-8f30-9c2a7e1b6d54", - "State": "Successful", - "ProcessName": "CustomerHubSync", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-18T02:15:03.940Z", - "EndTime": "2026-07-18T02:16:22.410Z", - "CreationTime": "2026-07-18T02:15:03.940Z" - }, - { - "Key": "3f6b0d29-7a15-4c38-9e02-4d8c1a5f9b37", - "State": "Successful", - "ProcessName": "CustomerHubSync", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-17T02:15:04.220Z", - "EndTime": "2026-07-17T02:16:19.980Z", - "CreationTime": "2026-07-17T02:15:04.220Z" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/78f94b6048.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/78f94b6048.json deleted file mode 100644 index f71a080fc8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/78f94b6048.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/README.md deleted file mode 100644 index a25ec833ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# UIAutomationNext Use Browser — Cannot Communicate With the Browser (Runtime) - -Runtime troubleshooting scenario for `UiPath.UIAutomationNext.Activities.NApplicationCard` (the modern -Use Application/Browser container) in a **browser** configuration. - -## What this scenario exercises - -An unattended job faults on the `Use Browser` scope with `Cannot communicate with the browser.` The -process ran clean on the same schedule and machine for weeks and the automation was not changed. The -agent must recognize this as a **broken browser-automation channel** (the browser is up, but the UiPath -extension / native-messaging bridge cannot carry commands) rather than a launch failure -(`ApplicationOpenException`) or a not-found (`ApplicationNotFoundException`), and prescribe the -extension / native-messaging / browser↔package-version remediation. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` with a `Use Browser` `NApplicationCard` (Chrome) wrapping `NTypeInto` / `NClick`; current, aligned package pins (no design-time skew) | -| `data/m/r/*.json` | faulted-job fixtures: `or folders/jobs list/get/logs/history/traces`. `job-get` + `job-logs` carry `Cannot communicate with the browser.` on the `NApplicationCard`; `jobs-list` shows the same process succeeding on the two prior nights (unchanged automation → environmental change) | -| `manifest.json` | `docsai ask` passthrough + job/folder rules + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the broken browser-automation channel and the remediation branches (UiPath extension - enabled/allowed-in-incognito, native-messaging host not blocked by antivirus/policy, browser↔package - version alignment), and ruled out launch-failure / not-found. - -Playbook: `references/activity-packages/ui-automation/playbooks/browser-communication-failed.md`. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/e2a5346a88.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/e2a5346a88.json deleted file mode 100644 index dc7fc46027..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/e2a5346a88.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T02:15:41.700Z", - "Message": "Use CustomerHub portal: Cannot communicate with the browser.\r\nin Main.xaml\r\n at NApplicationCard \"Use CustomerHub portal\"\r\n at Sequence \"Main Sequence\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.UiAutomationExecutionException: Cannot communicate with the browser.\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.d__.MoveNext()" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-19T02:15:05.020Z", - "Message": "CustomerHubSync started" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-19T02:15:04.510Z", - "Message": "CustomerHubSync execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/manifest.json deleted file mode 100644 index 1aac3ff25e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "78f94b6048.json" - }, - { - "match": "or jobs list", - "file": "6059da6ae1.json" - }, - { - "match": "or jobs logs", - "file": "e2a5346a88.json" - }, - { - "match": "or jobs history", - "file": "244ab94cfe.json" - }, - { - "match": "or jobs traces", - "file": "3672254a58.json" - }, - { - "match": "or jobs get", - "file": "464db476bb.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/uip-fixture.json new file mode 100644 index 0000000000..b8e282aaae --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7d2e5a10-9c34-4f62-b18a-3e6f0c9d2b47\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CustomerHubSync\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-19T02:15:04.110Z\",\r\n \"EndTime\": \"2026-07-19T02:15:41.870Z\",\r\n \"CreationTime\": \"2026-07-19T02:15:04.110Z\"\r\n },\r\n {\r\n \"Key\": \"5a9c1e83-2d47-4b16-8f30-9c2a7e1b6d54\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"CustomerHubSync\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-18T02:15:03.940Z\",\r\n \"EndTime\": \"2026-07-18T02:16:22.410Z\",\r\n \"CreationTime\": \"2026-07-18T02:15:03.940Z\"\r\n },\r\n {\r\n \"Key\": \"3f6b0d29-7a15-4c38-9e02-4d8c1a5f9b37\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"CustomerHubSync\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-17T02:15:04.220Z\",\r\n \"EndTime\": \"2026-07-17T02:16:19.980Z\",\r\n \"CreationTime\": \"2026-07-17T02:15:04.220Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T02:15:41.700Z\",\r\n \"Message\": \"Use CustomerHub portal: Cannot communicate with the browser.\\r\\nin Main.xaml\\r\\n at NApplicationCard \\\"Use CustomerHub portal\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.UiAutomationExecutionException: Cannot communicate with the browser.\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.d__.MoveNext()\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T02:15:05.020Z\",\r\n \"Message\": \"CustomerHubSync started\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T02:15:04.510Z\",\r\n \"Message\": \"CustomerHubSync execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-19T02:15:41.870Z\",\r\n \"JobId\": 66540211\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-19T02:15:04.640Z\",\r\n \"JobId\": 66540211\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7d2e5a10-9c34-4f62-b18a-3e6f0c9d2b47\",\r\n \"StartTime\": \"2026-07-19T02:15:04.110Z\",\r\n \"EndTime\": \"2026-07-19T02:15:41.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Use CustomerHub portal: Cannot communicate with the browser.\\r\\n\\r\\nin Main.xaml\\r\\n at NApplicationCard \\\"Use CustomerHub portal\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.UiAutomationExecutionException: Cannot communicate with the browser.\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.d__.MoveNext()\",\r\n \"CreationTime\": \"2026-07-19T02:15:04.110Z\",\r\n \"ReleaseName\": \"CustomerHubSync\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-19T02:15:41.870Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7d2e5a109c344f62b18a3e6f0c9d2b47\",\r\n \"Id\": 66540211\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/task.yaml index 2ec1045345..9185109e8e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-browser-communication-failed/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/README.md new file mode 100644 index 0000000000..cc84cbe49c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/README.md @@ -0,0 +1,52 @@ +# Uia Click No-Op (Simulate on Java) — Synthetic Scenario + +A modern Click (`NClick`) with `InteractionMode=Simulate` targets a **Java** +desktop app, where Simulate is unsupported. The click reports Successful while +the button is never actuated: the job ends `Successful` with zero Error logs and +the claim status stays `Draft`. This is the no-signature counterpart to the +Verify-Execution-failure case (nothing threw because no Verify was configured). + +This exercises the new +`skills/uipath-troubleshoot/references/activity-packages/ui-automation/playbooks/click-silent-no-op.md` +playbook and the no-signature routing path. + +## What the agent must uncover + +**Root Cause:** `InteractionMode=Simulate` on a Java target is a documented +silent no-op; with no Verify Execution, the miss never faults. **Fix:** switch +the Click's input method to Hardware Events (and add a real Verify target to +harden). See `RESOLUTION.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (Java scope, Simulate Click, no Verify) | + +Synthetic (not a faithful replay): fixtures are hand-authored so the root cause +is confirmable from runtime evidence (status still Draft) plus the workflow +source (Simulate input on a Java target). The agent-visible surface +(`process/` + `data/`) carries no diagnosis hints — the input-method/target-tech +knowledge lives in the skill's playbook, not in a planted fixture. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). +- Agent reached the same root cause + fix as `RESOLUTION.md` (`llm_judge`, threshold 0.7). + +## Running + +```bash +SKILLS_REPO_PATH="C:\\Work\\UiPath" .venv/Scripts/coder-eval.exe run \ + tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/task.yaml \ + -e experiments/default.yaml --repeats 3 -j 3 -v +``` + +Requires `SKILLS_REPO_PATH` and a tilde-free `TMPDIR/TEMP/TMP` (e.g. `C:/cetmp`) +on Windows. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/3dd609b65e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/3dd609b65e.json deleted file mode 100644 index e844db9e6d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/3dd609b65e.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", - "State": "Successful", - "ProcessName": "ClaimsSubmit", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-15T08:12:03.101Z", - "EndTime": "2026-07-15T08:12:41.552Z", - "CreationTime": "2026-07-15T08:12:02.400Z" - }, - { - "Key": "5aa17f60-9c31-4d02-8e4a-71b0c2d3e4f5", - "State": "Successful", - "ProcessName": "ClaimsSubmit", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-14T08:10:55.700Z", - "EndTime": "2026-07-14T08:11:34.210Z", - "CreationTime": "2026-07-14T08:10:55.100Z" - }, - { - "Key": "9d2b44e1-6f80-4c73-a15e-2b3c4d5e6f70", - "State": "Successful", - "ProcessName": "InvoiceExport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-07-13T15:02:11.000Z", - "EndTime": "2026-07-13T15:03:48.900Z", - "CreationTime": "2026-07-13T15:02:10.400Z" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/636b8bd298.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/636b8bd298.json deleted file mode 100644 index 9fbd429718..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/636b8bd298.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "Message": "ClaimsSubmit execution started", - "TimeStamp": "2026-07-15T08:12:03.220Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "ClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Claims - Java execution started", - "TimeStamp": "2026-07-15T08:12:05.010Z", - "RobotName": "svc_rpa", - "ProcessName": "ClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Click 'Submit Claim' execution started", - "TimeStamp": "2026-07-15T08:12:06.480Z", - "RobotName": "svc_rpa", - "ProcessName": "ClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Click 'Submit Claim' execution ended", - "TimeStamp": "2026-07-15T08:12:06.900Z", - "RobotName": "svc_rpa", - "ProcessName": "ClaimsSubmit" - }, - { - "Level": "Trace", - "Message": "Get Text 'Claim Status' execution ended", - "TimeStamp": "2026-07-15T08:12:07.640Z", - "RobotName": "svc_rpa", - "ProcessName": "ClaimsSubmit" - }, - { - "Level": "Information", - "Message": "Claim status after submit: Draft", - "TimeStamp": "2026-07-15T08:12:07.700Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "ClaimsSubmit" - }, - { - "Level": "Information", - "Message": "ClaimsSubmit execution ended", - "TimeStamp": "2026-07-15T08:12:41.400Z", - "RobotName": "svc_rpa", - "HostMachineName": "MOCK-HOST", - "ProcessName": "ClaimsSubmit" - } - ], - "Pagination": { - "Returned": 7, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/README.md deleted file mode 100644 index cc38f8eeed..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Uia Click No-Op (Simulate on Java) — Synthetic Scenario - -A modern Click (`NClick`) with `InteractionMode=Simulate` targets a **Java** -desktop app, where Simulate is unsupported. The click reports Successful while -the button is never actuated: the job ends `Successful` with zero Error logs and -the claim status stays `Draft`. This is the no-signature counterpart to the -Verify-Execution-failure case (nothing threw because no Verify was configured). - -This exercises the new -`skills/uipath-troubleshoot/references/activity-packages/ui-automation/playbooks/click-silent-no-op.md` -playbook and the no-signature routing path. - -## What the agent must uncover - -**Root Cause:** `InteractionMode=Simulate` on a Java target is a documented -silent no-op; with no Verify Execution, the miss never faults. **Fix:** switch -the Click's input method to Hardware Events (and add a real Verify target to -harden). See `RESOLUTION.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (Java scope, Simulate Click, no Verify) | -| `data/m/r/*.json` | canned `uip` CLI stdout: folders, jobs list/get, Error logs (empty), Info logs (status still Draft) | -| `data/m/r/manifest.json` | dispatch table mapping each command pattern to its fixture; `docsai ask` proxies to the real CLI (passthrough) | - -Synthetic (not a faithful replay): fixtures are hand-authored so the root cause -is confirmable from runtime evidence (status still Draft) plus the workflow -source (Simulate input on a Java target). The agent-visible surface -(`process/` + `data/`) carries no diagnosis hints — the input-method/target-tech -knowledge lives in the skill's playbook, not in a planted fixture. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). -- Agent reached the same root cause + fix as `RESOLUTION.md` (`llm_judge`, threshold 0.7). - -## Running - -```bash -SKILLS_REPO_PATH="C:\\Work\\UiPath" .venv/Scripts/coder-eval.exe run \ - tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/task.yaml \ - -e experiments/default.yaml --repeats 3 -j 3 -v -``` - -Requires `SKILLS_REPO_PATH` and a tilde-free `TMPDIR/TEMP/TMP` (e.g. `C:/cetmp`) -on Windows. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/b7a0b7b0c0.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/b7a0b7b0c0.json deleted file mode 100644 index 888f62f1f5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/b7a0b7b0c0.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Error", - "Code": "JobHealingData", - "Message": "Healing Agent is not enabled for this job. No recovery data is available.", - "Data": null -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/c0e1024460.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/c0e1024460.json deleted file mode 100644 index 385e69c9cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/c0e1024460.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071", - "DisplayName": "Finance", - "FullyQualifiedName": "Finance", - "ProvisionType": "Manual", - "PermissionModel": "InheritFromTenant", - "Id": 734512 - }, - { - "Key": "c0ffee11-2233-4455-6677-8899aabbccdd", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "ProvisionType": "Automatic", - "PermissionModel": "InheritFromTenant", - "Id": 1500287 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/c5eda10ee1.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/c5eda10ee1.json deleted file mode 100644 index 26238d5ebb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/c5eda10ee1.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 100, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/d4fa9befad.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/d4fa9befad.json deleted file mode 100644 index a8d3e66219..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/d4fa9befad.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", - "StartTime": "2026-07-15T08:12:03.101Z", - "EndTime": "2026-07-15T08:12:41.552Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", - "Info": "Job completed", - "CreationTime": "2026-07-15T08:12:02.400Z", - "ReleaseName": "ClaimsSubmit", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 30991244, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 734512, - "FolderKey": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svc_rpa", - "OrchestratorUserIdentity": "rpa-service@contoso.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-15T08:12:41.552Z", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 88123471 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/manifest.json deleted file mode 100644 index afbf2c63e1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/m/r/manifest.json +++ /dev/null @@ -1,37 +0,0 @@ -{ - "version": 2, - "rules": [ - { - "match": "or folders list", - "file": "c0e1024460.json" - }, - { - "match": "or jobs list", - "file": "3dd609b65e.json" - }, - { - "match": "or jobs get 7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", - "file": "d4fa9befad.json" - }, - { - "match": "or jobs logs 7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44 --level Error", - "file": "c5eda10ee1.json" - }, - { - "match": "or jobs logs 7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", - "file": "636b8bd298.json" - }, - { - "match": "or jobs healing-data 7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", - "file": "b7a0b7b0c0.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/uip-fixture.json new file mode 100644 index 0000000000..a46a8721db --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/data/uip-fixture.json @@ -0,0 +1,75 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44\",\r\n \"StartTime\": \"2026-07-15T08:12:03.101Z\",\r\n \"EndTime\": \"2026-07-15T08:12:41.552Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-07-15T08:12:02.400Z\",\r\n \"ReleaseName\": \"ClaimsSubmit\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 30991244,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 734512,\r\n \"FolderKey\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_rpa\",\r\n \"OrchestratorUserIdentity\": \"rpa-service@contoso.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-15T08:12:41.552Z\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 88123471\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"ClaimsSubmit execution started\",\r\n \"TimeStamp\": \"2026-07-15T08:12:03.220Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Claims - Java execution started\",\r\n \"TimeStamp\": \"2026-07-15T08:12:05.010Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Click 'Submit Claim' execution started\",\r\n \"TimeStamp\": \"2026-07-15T08:12:06.480Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Click 'Submit Claim' execution ended\",\r\n \"TimeStamp\": \"2026-07-15T08:12:06.900Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"Message\": \"Get Text 'Claim Status' execution ended\",\r\n \"TimeStamp\": \"2026-07-15T08:12:07.640Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"Claim status after submit: Draft\",\r\n \"TimeStamp\": \"2026-07-15T08:12:07.700Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"Message\": \"ClaimsSubmit execution ended\",\r\n \"TimeStamp\": \"2026-07-15T08:12:41.400Z\",\r\n \"RobotName\": \"svc_rpa\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"ClaimsSubmit\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 7,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data", + "7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Error\",\r\n \"Code\": \"JobHealingData\",\r\n \"Message\": \"Healing Agent is not enabled for this job. No recovery data is available.\",\r\n \"Data\": null\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071\",\r\n \"DisplayName\": \"Finance\",\r\n \"FullyQualifiedName\": \"Finance\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"InheritFromTenant\",\r\n \"Id\": 734512\r\n },\r\n {\r\n \"Key\": \"c0ffee11-2233-4455-6677-8899aabbccdd\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"ProvisionType\": \"Automatic\",\r\n \"PermissionModel\": \"InheritFromTenant\",\r\n \"Id\": 1500287\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7c4e1a9b-2d6f-4a81-b3e0-9f5c1d8a2e44\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ClaimsSubmit\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-15T08:12:03.101Z\",\r\n \"EndTime\": \"2026-07-15T08:12:41.552Z\",\r\n \"CreationTime\": \"2026-07-15T08:12:02.400Z\"\r\n },\r\n {\r\n \"Key\": \"5aa17f60-9c31-4d02-8e4a-71b0c2d3e4f5\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ClaimsSubmit\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-14T08:10:55.700Z\",\r\n \"EndTime\": \"2026-07-14T08:11:34.210Z\",\r\n \"CreationTime\": \"2026-07-14T08:10:55.100Z\"\r\n },\r\n {\r\n \"Key\": \"9d2b44e1-6f80-4c73-a15e-2b3c4d5e6f70\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"InvoiceExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-07-13T15:02:11.000Z\",\r\n \"EndTime\": \"2026-07-13T15:03:48.900Z\",\r\n \"CreationTime\": \"2026-07-13T15:02:10.400Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/task.yaml index ad2c40c655..21cde5c3af 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-click-noop-simulate-tech/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./m to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/README.md new file mode 100644 index 0000000000..63ec8f6f9d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/README.md @@ -0,0 +1,61 @@ +# UIAutomationNext Failure - MissingMethodException / Dependency Version Conflict + +This scenario reproduces a **design-time / validation** failure of a +UIAutomationNext (`NClick` / modern Click) project: opening or validating the +process throws `System.MissingMethodException` — "Method not found: 'Void +UiPath.UIAutomationNext.Activities...'" — because `project.json` pins +`UiPath.UIAutomation.Activities` out of step with `UiPath.System.Activities` +and the installed Studio. The process has **never run** — there are no +Orchestrator jobs. + +## What this scenario uncovers + +**Root Cause:** `project.json` pins `UiPath.UIAutomation.Activities [24.10.9]` +against `UiPath.System.Activities [22.4.4]` and `studioVersion 22.4.3.0`. The +UI Automation package was bumped in isolation; the resolved +`UiPath.UIAutomationNext(.Activities)` runtime is older than the package +expects, so the activities' method lookup fails at bind time and Studio +throws `MissingMethodException`. The diagnosis comes from the **project +source (the mismatched dependency set)**, not job evidence. + +This maps to: +`references/activity-packages/ui-automation/playbooks/dependency-version-conflict.md` +(**Signature A** — Method not found / MissingMethodException). + +The correct agent behavior is to reason from `project.json` (the version skew +across UI Automation vs System.Activities vs Studio) rather than chasing +Orchestrator jobs, match the dependency-version-conflict playbook, and +recommend aligning **all** foundational packages to a mutually compatible +line via Manage Packages — explicitly NOT the **Signature B** runtime +"Could not load file or assembly" fix (clean `%userprofile%\.nuget\packages` +/ republish), which does not apply because the assembly loads and the failure +reproduces in Studio at design time. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UIAutomationNext project (`NApplicationCard` + `NClick`) pinning `UiPath.UIAutomation.Activities [24.10.9]` vs `UiPath.System.Activities [22.4.4]` with `studioVersion 22.4.3.0` | + +> **Note on fixtures.** Like the Word `replace-text-version-mismatch` +> scenario, this has **no faulted job** by design — it is a design-time +> validation error, so `or jobs list` returns empty and the agent must +> diagnose from `project.json` + the Studio version in the prompt. It tests +> whether the agent correctly distinguishes an in-project version conflict +> (Signature A) from the robot-side assembly-load failure (Signature B). + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `dependency-version-conflict.md` (Signature A) +- Agent identified a UIAutomationNext dependency version conflict inside + `project.json` (UI Automation package bumped out of step with + System.Activities / Studio 2022.4) as the design-time cause, reasoning from + the project source, and recommended aligning all foundational packages to a + compatible line via Manage Packages — without confusing it with the + Signature B runtime "cannot load assembly" cache/republish fix or + fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/2e37ebc77e.json deleted file mode 100644 index f79d8dcce7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e", - "Name": "Onboarding Dev", - "Path": "Onboarding Dev", - "Description": "Development folder for onboarding automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/README.md deleted file mode 100644 index c49ef3dcf5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# UIAutomationNext Failure - MissingMethodException / Dependency Version Conflict - -This scenario reproduces a **design-time / validation** failure of a -UIAutomationNext (`NClick` / modern Click) project: opening or validating the -process throws `System.MissingMethodException` — "Method not found: 'Void -UiPath.UIAutomationNext.Activities...'" — because `project.json` pins -`UiPath.UIAutomation.Activities` out of step with `UiPath.System.Activities` -and the installed Studio. The process has **never run** — there are no -Orchestrator jobs. - -## What this scenario uncovers - -**Root Cause:** `project.json` pins `UiPath.UIAutomation.Activities [24.10.9]` -against `UiPath.System.Activities [22.4.4]` and `studioVersion 22.4.3.0`. The -UI Automation package was bumped in isolation; the resolved -`UiPath.UIAutomationNext(.Activities)` runtime is older than the package -expects, so the activities' method lookup fails at bind time and Studio -throws `MissingMethodException`. The diagnosis comes from the **project -source (the mismatched dependency set)**, not job evidence. - -This maps to: -`references/activity-packages/ui-automation/playbooks/dependency-version-conflict.md` -(**Signature A** — Method not found / MissingMethodException). - -The correct agent behavior is to reason from `project.json` (the version skew -across UI Automation vs System.Activities vs Studio) rather than chasing -Orchestrator jobs, match the dependency-version-conflict playbook, and -recommend aligning **all** foundational packages to a mutually compatible -line via Manage Packages — explicitly NOT the **Signature B** runtime -"Could not load file or assembly" fix (clean `%userprofile%\.nuget\packages` -/ republish), which does not apply because the assembly loads and the failure -reproduces in Studio at design time. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` dispatcher | shared from `../../_shared/mock_template/` | -| `process/` | hand-authored UIAutomationNext project (`NApplicationCard` + `NClick`) pinning `UiPath.UIAutomation.Activities [24.10.9]` vs `UiPath.System.Activities [22.4.4]` with `studioVersion 22.4.3.0` | -| `data/m/r/*.json` | folders list + **empty** job lists (the process never ran) | -| `data/m/r/manifest.json` | dispatch table (folders, empty jobs, `docsai ask` passthrough, permissive `[]` fallback) | - -> **Note on fixtures.** Like the Word `replace-text-version-mismatch` -> scenario, this has **no faulted job** by design — it is a design-time -> validation error, so `or jobs list` returns empty and the agent must -> diagnose from `project.json` + the Studio version in the prompt. It tests -> whether the agent correctly distinguishes an in-project version conflict -> (Signature A) from the robot-side assembly-load failure (Signature B). - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `dependency-version-conflict.md` (Signature A) -- Agent identified a UIAutomationNext dependency version conflict inside - `project.json` (UI Automation package bumped out of step with - System.Activities / Studio 2022.4) as the design-time cause, reasoning from - the project source, and recommended aligning all foundational packages to a - compatible line via Manage Packages — without confusing it with the - Signature B runtime "cannot load assembly" cache/republish fix or - fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/b41a034ebf.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/b41a034ebf.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/b41a034ebf.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/manifest.json deleted file mode 100644 index af17c2f37d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatcher fixture set for this scenario; not investigation evidence. Provides a folder list and empty job lists (this process has no Orchestrator runs).", - "rules": [ - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e\" --state Faulted --output json", - "file": "b41a034ebf.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e --state Faulted --output json", - "file": "b41a034ebf.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e\" --output json", - "file": "b41a034ebf.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e --output json", - "file": "b41a034ebf.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands (no jobs exist for this design-time crash)." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/uip-fixture.json new file mode 100644 index 0000000000..14502e4f3c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/data/uip-fixture.json @@ -0,0 +1,51 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-4b8c-9d0e-1f2a3b4c5d6e\",\r\n \"Name\": \"Onboarding Dev\",\r\n \"Path\": \"Onboarding Dev\",\r\n \"Description\": \"Development folder for onboarding automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/task.yaml index e731f20277..23fca1b569 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-dependency-version-conflict/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/README.md new file mode 100644 index 0000000000..de558dbd58 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/README.md @@ -0,0 +1,35 @@ +# Healing Agent Disabled / No Consumption — Faithful Replay + +Replays a real `uipath-troubleshoot` investigation against a `uip` CLI mock. The fixtures are the verbatim sub-agent CLI responses captured from that session (PII scrubbed). + +## What the original session uncovered + +A user asked whether the Healing Agent ran on their **last job in folder `Shared`**, whether it consumed any AI units/credits, and what the `healing-agent` log line meant. The most-recent job (`BlankProcess2`, Attended/StudioPro) had **Faulted** on an unrelated UI Automation `NodeNotFoundException`. Healing Agent was **disabled** for the run (`AutopilotForRobots.Enabled=false`, `HealingEnabled=false`); the `"Healing agent is disabled for the current job."` Info log is a benign config-read; the healing-data archive is an empty 22-byte ZIP. Conclusion: **HA did not run and consumed nothing** — this is Surface 2 (the config-read log line) of the `healing-agent-orch-issues` playbook. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +No `process/` snapshot — the source project was not part of the session (the investigation was CLI-driven). + +## Success criteria + +Scores the **conclusion**, not the trajectory (per `../CLAUDE.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- LLM judge (vs `RESOLUTION.md`): HA did not run, zero consumption, the log line is informational. + +## Re-running the extraction + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation <.local/investigations dir> --transcript \ + --scenario-name healing-agent-disabled-no-consumption --apply +``` + +Then repopulate any stub fixtures from the investigation's `raw/` responses and rewrite the manifest matches as unquoted substrings. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/0f27502236.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/0f27502236.json deleted file mode 100644 index 809b0853c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/0f27502236.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "StartTime": "2026-06-02T07:30:38.093Z", - "EndTime": "2026-06-02T07:31:37.367Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "Info": "Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive\r\n\r\nin Main.xaml\r\n at NClick \"Click 'I'm Feeling Lucky'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-02T07:30:38.093Z", - "ReleaseName": "BlankProcess2", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20708252, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-02T07:31:37.367Z", - "JobError": {}, - "ErrorCode": "Robot", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 65777592 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/3b44f9e096.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/3b44f9e096.json deleted file mode 100644 index 9cd5e0b98e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/3b44f9e096.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-02T07:31:36.167Z", - "Message": "BlankProcess2 execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-02T07:31:35.595Z", - "Message": "Click 'I'm Feeling Lucky': Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive\r\nin Main.xaml\r\n at NClick \"Click 'I'm Feeling Lucky'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T07:31:03.605Z", - "Message": "Healing agent is disabled for the current job." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-02T07:31:02.682Z", - "Message": "BlankProcess2 execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/4fd33df393.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/4fd33df393.json deleted file mode 100644 index 208d888e78..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/4fd33df393.json +++ /dev/null @@ -1,418 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T07:30:38.093Z", - "EndTime": "2026-06-02T07:31:37.367Z", - "CreationTime": "2026-06-02T07:30:38.093Z" - }, - { - "Key": "68d80da7-719f-44cd-b7d6-94be679f55d4", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:52:06.580Z", - "EndTime": "2026-06-02T06:52:40.293Z", - "CreationTime": "2026-06-02T06:52:06.580Z" - }, - { - "Key": "3975787c-87d9-4927-bc7c-a19165e06035", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:42:21.737Z", - "EndTime": "2026-06-02T06:42:56.063Z", - "CreationTime": "2026-06-02T06:42:21.737Z" - }, - { - "Key": "b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d", - "State": "Pending", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-29T12:28:56.660Z" - }, - { - "Key": "ccc3e841-bdcf-4501-9d30-f112f098f630", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T13:09:18.353Z", - "EndTime": "2026-05-28T13:10:10.310Z", - "CreationTime": "2026-05-28T12:48:07.547Z" - }, - { - "Key": "cb00355c-3835-4c18-b37e-af5c7609c599", - "State": "Successful", - "ProcessName": "NewBlankTask", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T10:28:07.327Z", - "EndTime": "2026-05-28T10:29:16.540Z", - "CreationTime": "2026-05-28T08:48:18.580Z" - }, - { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T08:00:50.870Z", - "EndTime": "2026-05-28T08:01:02.417Z", - "CreationTime": "2026-05-27T16:24:24.603Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-26T08:48:20.617Z", - "EndTime": "2026-05-26T08:48:20.617Z", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 29, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/53cb28700c.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/53cb28700c.json deleted file mode 100644 index d1132089af..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/53cb28700c.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHealingData", - "Data": { - "JobKey": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "FileName": "", - "Size": 22, - "ExportId": 11700 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/README.md deleted file mode 100644 index 803660462c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/README.md +++ /dev/null @@ -1,34 +0,0 @@ -# Healing Agent Disabled / No Consumption — Faithful Replay - -Replays a real `uipath-troubleshoot` investigation against a `uip` CLI mock. The fixtures are the verbatim sub-agent CLI responses captured from that session (PII scrubbed). - -## What the original session uncovered - -A user asked whether the Healing Agent ran on their **last job in folder `Shared`**, whether it consumed any AI units/credits, and what the `healing-agent` log line meant. The most-recent job (`BlankProcess2`, Attended/StudioPro) had **Faulted** on an unrelated UI Automation `NodeNotFoundException`. Healing Agent was **disabled** for the run (`AutopilotForRobots.Enabled=false`, `HealingEnabled=false`); the `"Healing agent is disabled for the current job."` Info log is a benign config-read; the healing-data archive is an empty 22-byte ZIP. Conclusion: **HA did not run and consumed nothing** — this is Surface 2 (the config-read log line) of the `healing-agent-orch-issues` playbook. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout from the session (`jobs get` / `jobs list` / `jobs logs` repopulated from the recorded raw responses — the generator's compound-command backfill produced stubs) | -| `fixtures/mocks/responses/manifest.json` | substring dispatch table (unquoted keys) → recorded fixtures | - -No `process/` snapshot — the source project was not part of the session (the investigation was CLI-driven). - -## Success criteria - -Scores the **conclusion**, not the trajectory (per `../CLAUDE.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- LLM judge (vs `RESOLUTION.md`): HA did not run, zero consumption, the log line is informational. - -## Re-running the extraction - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation <.local/investigations dir> --transcript \ - --scenario-name healing-agent-disabled-no-consumption --apply -``` - -Then repopulate any stub fixtures from the investigation's `raw/` responses and rewrite the manifest matches as unquoted substrings. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/a728a49add.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/a728a49add.json deleted file mode 100644 index 719ff2d2fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/a728a49add.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "LicenseInfo", - "Data": { - "IsExpired": false, - "SubscriptionPlan": "ENTERPRISE", - "Allowed": { - "Headless": 0, - "Flow": 0, - "Development": 0, - "StudioX": 0, - "StudioPro": 0, - "NonProduction": 0, - "Unattended": 1, - "Hosting": 0, - "Local": 0, - "AgentService": 0, - "CaseManagement": 0, - "AppTest": 0, - "PerformanceTest": 0, - "Attended": 0, - "ProcessOrchestration": 0, - "BusinessRule": 0, - "TestAutomation": 1 - }, - "Used": { - "Headless": 0, - "Flow": 0, - "Development": 0, - "StudioX": 0, - "StudioPro": 0, - "NonProduction": 0, - "Unattended": 1, - "Hosting": 0, - "Local": 0, - "AgentService": 0, - "CaseManagement": 0, - "AppTest": 0, - "PerformanceTest": 0, - "Attended": 0, - "ProcessOrchestration": 0, - "BusinessRule": 0, - "TestAutomation": 0 - }, - "LicensedFeatures": [] - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/manifest.json deleted file mode 100644 index 1648031f67..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/m/r/manifest.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "version": 2, - "_doc": "Replay of a Healing-Agent 'is this expected / did it consume units' investigation. Matches are plain substrings (first match wins) and intentionally UNQUOTED so they match whether or not the agent quotes the job/folder key. jobs-get / jobs-list / jobs-logs fixtures were repopulated from the recorded raw responses (the generator's compound-command backfill produced stubs).", - "rules": [ - { - "match": "or folders list", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "4fd33df393.json" - }, - { - "match": "or jobs get ", - "file": "0f27502236.json" - }, - { - "match": "or jobs logs ", - "file": "3b44f9e096.json" - }, - { - "match": "or jobs healing-data ", - "file": "53cb28700c.json" - }, - { - "match": "or licenses info", - "file": "a728a49add.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/uip-fixture.json new file mode 100644 index 0000000000..4e6128c996 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/data/uip-fixture.json @@ -0,0 +1,71 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\"\r\n },\r\n {\r\n \"Key\": \"68d80da7-719f-44cd-b7d6-94be679f55d4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:52:06.580Z\",\r\n \"EndTime\": \"2026-06-02T06:52:40.293Z\",\r\n \"CreationTime\": \"2026-06-02T06:52:06.580Z\"\r\n },\r\n {\r\n \"Key\": \"3975787c-87d9-4927-bc7c-a19165e06035\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:42:21.737Z\",\r\n \"EndTime\": \"2026-06-02T06:42:56.063Z\",\r\n \"CreationTime\": \"2026-06-02T06:42:21.737Z\"\r\n },\r\n {\r\n \"Key\": \"b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-29T12:28:56.660Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"cb00355c-3835-4c18-b37e-af5c7609c599\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NewBlankTask\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T10:28:07.327Z\",\r\n \"EndTime\": \"2026-05-28T10:29:16.540Z\",\r\n \"CreationTime\": \"2026-05-28T08:48:18.580Z\"\r\n },\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T08:00:50.870Z\",\r\n \"EndTime\": \"2026-05-28T08:01:02.417Z\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 29,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"Info\": \"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive\\r\\n\\r\\nin Main.xaml\\r\\n at NClick \\\"Click 'I'm Feeling Lucky'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"ReleaseName\": \"BlankProcess2\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20708252,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 65777592\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T07:31:36.167Z\",\r\n \"Message\": \"BlankProcess2 execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-02T07:31:35.595Z\",\r\n \"Message\": \"Click 'I'm Feeling Lucky': Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive\\r\\nin Main.xaml\\r\\n at NClick \\\"Click 'I'm Feeling Lucky'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T07:31:03.605Z\",\r\n \"Message\": \"Healing agent is disabled for the current job.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-02T07:31:02.682Z\",\r\n \"Message\": \"BlankProcess2 execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHealingData\",\r\n \"Data\": {\r\n \"JobKey\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"FileName\": \"\",\r\n \"Size\": 22,\r\n \"ExportId\": 11700\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "licenses", + "info" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LicenseInfo\",\r\n \"Data\": {\r\n \"IsExpired\": false,\r\n \"SubscriptionPlan\": \"ENTERPRISE\",\r\n \"Allowed\": {\r\n \"Headless\": 0,\r\n \"Flow\": 0,\r\n \"Development\": 0,\r\n \"StudioX\": 0,\r\n \"StudioPro\": 0,\r\n \"NonProduction\": 0,\r\n \"Unattended\": 1,\r\n \"Hosting\": 0,\r\n \"Local\": 0,\r\n \"AgentService\": 0,\r\n \"CaseManagement\": 0,\r\n \"AppTest\": 0,\r\n \"PerformanceTest\": 0,\r\n \"Attended\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"BusinessRule\": 0,\r\n \"TestAutomation\": 1\r\n },\r\n \"Used\": {\r\n \"Headless\": 0,\r\n \"Flow\": 0,\r\n \"Development\": 0,\r\n \"StudioX\": 0,\r\n \"StudioPro\": 0,\r\n \"NonProduction\": 0,\r\n \"Unattended\": 1,\r\n \"Hosting\": 0,\r\n \"Local\": 0,\r\n \"AgentService\": 0,\r\n \"CaseManagement\": 0,\r\n \"AppTest\": 0,\r\n \"PerformanceTest\": 0,\r\n \"Attended\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"BusinessRule\": 0,\r\n \"TestAutomation\": 0\r\n },\r\n \"LicensedFeatures\": []\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/task.yaml index 00a14c1c6c..0bfb5ab1c6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-disabled-no-consumption/task.yaml @@ -23,21 +23,10 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/README.md new file mode 100644 index 0000000000..2d0d2f723c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/README.md @@ -0,0 +1,40 @@ +# Healing Agent Invalid License — Faithful Replay + +This scenario replays a real UiPath diagnostic investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +A Faulted Orchestrator job for process **ERN** in folder **Shared** (job key `d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff`) failed with `UiPath.UIAutomationNext.Exceptions.NodeNotFoundException` at the **Click 'Simt că am noroc'** activity in `Google.xaml` — an authoring-time selector typo (`aria-label='Simt că am noroccccccccccc'` with nine extra trailing `c` characters vs. the live `aria-label='Simt că am noroc'` at 94% match). + +`AutopilotForRobots.Enabled=true` and `HealingEnabled=true` on the job, so Healing Agent was invoked — but it refused to run. The robot Error log at `2026-05-15T17:10:18.218Z` contains the deterministic line *"'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery."*, `uip or jobs healing-data` returned a 22-byte empty ZIP, and `uip or licenses info` confirmed `Allowed.AgentService=0` with `LicensedFeatures=[]` despite `SubscriptionPlan='ENTERPRISE'`. + +The match is **cause #1** in `healing-agent-no-license.md` — *No Healing Agent entitlement on the tenant*. RuntimeType=`StudioPro` (Non-Test set) means the requested pool is regular **Heals** (operation code `HealingAgent`), not Test Heals. Fix: acquire the Healing Agent Add-On for the tenant, or move to Flex Advanced / Unified Enterprise. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-diagnostics` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name healing-agent-invalid-license --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/055fede411.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/055fede411.json deleted file mode 100644 index 95e20e2d01..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/055fede411.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-05-15T17:10:18.563Z", - "JobId": 64312253 - }, - { - "State": "Running", - "Timestamp": "2026-05-15T17:09:44.497Z", - "JobId": 64312253 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/108714ddb6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/108714ddb6.json deleted file mode 100644 index e5161bc625..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/108714ddb6.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "Info": "Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\n\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-15T17:09:44.470Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20182479, - "EntryPointPath": "Google.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-15T17:10:18.527Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d2c90d73bcee4f9db9fcd37146b7f6ff", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 64312253 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/4d0dc464a8.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/4d0dc464a8.json deleted file mode 100644 index 0305af2123..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/4d0dc464a8.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-15T17:10:18.631Z", - "Message": "Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-15T17:10:18.218Z", - "Message": "'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/62e9b8f4c2.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/62e9b8f4c2.json deleted file mode 100644 index 836a5627a5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/62e9b8f4c2.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "folders", - "Description": "Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder.", - "Usage": "uip or folders [options] [command]", - "Arguments": [], - "Options": [ - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Subcommands": [ - { - "Name": "list [options]", - "Description": "List folders the current user has access to. Use --all to list all folders in the tenant (supports filtering and sorting)." - }, - { - "Name": "create [options] ", - "Description": "Create a new folder. Folders scope processes, jobs, and resources with separate permissions. Use --parent to nest inside an existing folder." - }, - { - "Name": "get [options] ", - "Description": "Get folder details by key (GUID) or path. Returns display name, key (GUID), path, type, and permissions. Use 'folders list' to find folder keys." - }, - { - "Name": "delete [options] ", - "Description": "Delete a folder by key (GUID) or path. Fails if the folder contains entities or user associations. Move or remove contents before deleting." - }, - { - "Name": "move [options] ", - "Description": "Move a folder to a new parent in the folder hierarchy. Provide the target parent folder path or key. Use --root to move to the top level." - }, - { - "Name": "edit [options] ", - "Description": "Edit folder properties (display name, description). Use 'folders get' to see current values before editing." - }, - { - "Name": "runtimes [options] ", - "Description": "List runtime allocations for a folder. Shows how many runtime slots of each type (Unattended, Attended, NonProduction, etc.) are total, connected, and available in the specified folder. Useful for checking capacity before starting jobs." - }, - { - "Name": "help [command]", - "Description": "display help for command" - } - ], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/6eb1e3ebb0.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/6eb1e3ebb0.json deleted file mode 100644 index bc4043342a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/6eb1e3ebb0.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderGet", - "Data": { - "Name": "Shared", - "ID": 1500287, - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Description": "No description", - "Path": "Shared", - "ParentID": "Root", - "FolderType": "Standard", - "IsPersonal": "No", - "ProvisionType": "Automatic", - "PermissionModel": "FineGrained", - "FeedType": "Processes" - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/947f74b847.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/947f74b847.json deleted file mode 100644 index e2e704b108..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/947f74b847.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "LicenseInfo", - "Data": { - "IsExpired": false, - "SubscriptionPlan": "ENTERPRISE", - "Allowed": { - "CaseManagement": 0, - "Flow": 0, - "StudioX": 0, - "TestAutomation": 1, - "Attended": 0, - "Hosting": 0, - "Unattended": 1, - "Development": 0, - "Headless": 0, - "AgentService": 0, - "StudioPro": 0, - "BusinessRule": 0, - "Agent": 0, - "NonProduction": 0, - "ProcessOrchestration": 0, - "AppTest": 0, - "PerformanceTest": 0 - }, - "Used": { - "CaseManagement": 0, - "Flow": 0, - "StudioX": 0, - "TestAutomation": 0, - "Attended": 0, - "Hosting": 0, - "Unattended": 1, - "Development": 0, - "Headless": 0, - "AgentService": 0, - "StudioPro": 0, - "BusinessRule": 0, - "Agent": 0, - "NonProduction": 0, - "ProcessOrchestration": 0, - "AppTest": 0, - "PerformanceTest": 0 - }, - "LicensedFeatures": [] - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/README.md deleted file mode 100644 index cbedea5e74..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Healing Agent Invalid License — Faithful Replay - -This scenario replays a real UiPath diagnostic investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -A Faulted Orchestrator job for process **ERN** in folder **Shared** (job key `d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff`) failed with `UiPath.UIAutomationNext.Exceptions.NodeNotFoundException` at the **Click 'Simt că am noroc'** activity in `Google.xaml` — an authoring-time selector typo (`aria-label='Simt că am noroccccccccccc'` with nine extra trailing `c` characters vs. the live `aria-label='Simt că am noroc'` at 94% match). - -`AutopilotForRobots.Enabled=true` and `HealingEnabled=true` on the job, so Healing Agent was invoked — but it refused to run. The robot Error log at `2026-05-15T17:10:18.218Z` contains the deterministic line *"'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery."*, `uip or jobs healing-data` returned a 22-byte empty ZIP, and `uip or licenses info` confirmed `Allowed.AgentService=0` with `LicensedFeatures=[]` despite `SubscriptionPlan='ENTERPRISE'`. - -The match is **cause #1** in `healing-agent-no-license.md` — *No Healing Agent entitlement on the tenant*. RuntimeType=`StudioPro` (Non-Test set) means the requested pool is regular **Heals** (operation code `HealingAgent`), not Test Heals. Fix: acquire the Healing Agent Add-On for the tenant, or move to Flex Advanced / Unified Enterprise. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-diagnostics` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name healing-agent-invalid-license --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/a9ae47f3f6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/a9ae47f3f6.json deleted file mode 100644 index 246ab6c89e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/a9ae47f3f6.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHealingData", - "Data": { - "JobKey": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "FileName": "", - "Size": 22, - "ExportId": 11659, - "_doc": "HA produced no recovery data — the on-disk ZIP at the agent's -o path would be 22 bytes (end-of-central-directory record only, zero entries). The mock does not materialize the file; the agent should rely on Size=22 here AND the 'No available license' Error log line from `or jobs logs --level Error` for the deterministic empty-ZIP signal per healing-agent-no-license.md step 3." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/be540c1e66.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/be540c1e66.json deleted file mode 100644 index 75f6a00710..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/be540c1e66.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z", - "ReleaseName": "ERN", - "Type": "Attended", - "RuntimeType": "StudioPro", - "HostMachineName": "MOCK-HOST", - "EntryPointPath": "Google.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "ErrorCode": "Robot", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 64312253 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/dca66526a6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/dca66526a6.json deleted file mode 100644 index a01acd422e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/dca66526a6.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "No description", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/manifest.json deleted file mode 100644 index 3691ce77c0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/m/r/manifest.json +++ /dev/null @@ -1,83 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of a UIA Healing-Agent-no-license investigation. Triage rules captured verbatim from the original session; license/log fixtures restored from the investigation's raw/ folder. Match patterns are kept permissive (no quote-sensitive matching) so the agent succeeds even if it formats CLI args slightly differently than the recording.", - "rules": [ - { - "match": "or licenses info", - "file": "947f74b847.json", - "_doc": "Deterministic discriminator for healing-agent-no-license.md cause #1: Allowed.AgentService=0 AND LicensedFeatures=[] with SubscriptionPlan='ENTERPRISE'. Catches any `or licenses info` variant the agent issues (with/without --output json)." - }, - { - "match": "or folders list", - "file": "dca66526a6.json", - "_doc": "Returns the Shared folder (Key=defb8e05-...) so the agent can resolve the folder for the faulted job. v1.0.4+ `or folders list` defaults to user-accessible folders (Personal/Solution/Standard)." - }, - { - "match": "or folders --help", - "file": "62e9b8f4c2.json", - "_doc": "Help text — included for completeness in case the agent explores command surface." - }, - { - "match": "or folders get", - "file": "6eb1e3ebb0.json", - "_doc": "Folder lookup for 'Shared' → returns Key=defb8e05-... and ID=1500287. Catches any `or folders get ` variant." - }, - { - "match": "or jobs list", - "file": "be540c1e66.json", - "_doc": "Catch-all: any `or jobs list` (with or without --state Faulted / --folder-key) returns the recorded faulted-job d2c90d73-... in folder Shared." - }, - { - "match": "or jobs get d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "file": "108714ddb6.json", - "_doc": "Authoritative job record: State=Faulted, AutopilotForRobots.Enabled=true, HealingEnabled=true, RuntimeType=StudioPro, JobError={}, ErrorCode=Robot. Matches any `or jobs get d2c90d73...` variant — quotes optional." - }, - { - "match": "or jobs logs d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "file": "4d0dc464a8.json", - "_doc": "Robot Error logs — contains the deterministic 'No available license / Agentic units to perform healing analysis and recovery' message at 17:10:18.218Z AND the NodeNotFoundException at 17:10:18.631Z for the same activity. The HA-refusal line is the primary signal per healing-agent-no-license.md step 1." - }, - { - "match": "or jobs healing-data d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "file": "a9ae47f3f6.json", - "_doc": "Marker showing Size=22 (empty ZIP, zero entries) — the corroborating signal per healing-agent-no-license.md step 3 that HA refused to run. The mock does not materialize the actual ZIP on disk; agents should treat Size in the response as authoritative." - }, - { - "match": "or jobs history d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "file": "055fede411.json", - "_doc": "Minimal Faulted→Running history. Not load-bearing for this playbook; included so the agent does not abort if it explores history." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai query strings vary between runs; proxy to the real `uip docsai ask` on the host. Dispatcher caches each unique query under responses/_cache/ for in-run reuse." - } - ], - "expected_calls": [ - { - "pattern": "or jobs get d2c90d73", - "min": 1, - "description": "Agent fetches the faulted job to confirm AutopilotForRobots.Enabled, HealingEnabled, RuntimeType, State=Faulted (playbook step 2)." - }, - { - "pattern": "or jobs logs d2c90d73", - "min": 1, - "description": "Agent fetches Error-level robot logs to find the 'No available license / Agentic units' line (playbook step 1, the deterministic signal)." - }, - { - "pattern": "or licenses info", - "min": 1, - "description": "Agent reads tenant license to discriminate the playbook cause (#1 missing entitlement vs #7 exhausted) — Allowed.AgentService=0, LicensedFeatures=[] confirms #1 (playbook step 5)." - }, - { - "pattern": "or jobs healing-data d2c90d73", - "min": 0, - "description": "Optional: corroborates with empty-ZIP marker (playbook step 3). Agent may skip this when the Error log + licenses info already deterministically identify cause #1." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/uip-fixture.json new file mode 100644 index 0000000000..77b2224580 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/data/uip-fixture.json @@ -0,0 +1,125 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "get", + "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"Info\": \"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20182479,\r\n \"EntryPointPath\": \"Google.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d2c90d73bcee4f9db9fcd37146b7f6ff\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 64312253\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-15T17:10:18.631Z\",\r\n \"Message\": \"Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-15T17:10:18.218Z\",\r\n \"Message\": \"'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data", + "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobHealingData\",\n \"Data\": {\n \"JobKey\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\n \"FileName\": \"\",\n \"Size\": 22,\n \"ExportId\": 11659\n }\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-15T17:10:18.563Z\",\r\n \"JobId\": 64312253\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-15T17:09:44.497Z\",\r\n \"JobId\": 64312253\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "licenses", + "info" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LicenseInfo\",\r\n \"Data\": {\r\n \"IsExpired\": false,\r\n \"SubscriptionPlan\": \"ENTERPRISE\",\r\n \"Allowed\": {\r\n \"CaseManagement\": 0,\r\n \"Flow\": 0,\r\n \"StudioX\": 0,\r\n \"TestAutomation\": 1,\r\n \"Attended\": 0,\r\n \"Hosting\": 0,\r\n \"Unattended\": 1,\r\n \"Development\": 0,\r\n \"Headless\": 0,\r\n \"AgentService\": 0,\r\n \"StudioPro\": 0,\r\n \"BusinessRule\": 0,\r\n \"Agent\": 0,\r\n \"NonProduction\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"AppTest\": 0,\r\n \"PerformanceTest\": 0\r\n },\r\n \"Used\": {\r\n \"CaseManagement\": 0,\r\n \"Flow\": 0,\r\n \"StudioX\": 0,\r\n \"TestAutomation\": 0,\r\n \"Attended\": 0,\r\n \"Hosting\": 0,\r\n \"Unattended\": 1,\r\n \"Development\": 0,\r\n \"Headless\": 0,\r\n \"AgentService\": 0,\r\n \"StudioPro\": 0,\r\n \"BusinessRule\": 0,\r\n \"Agent\": 0,\r\n \"NonProduction\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"AppTest\": 0,\r\n \"PerformanceTest\": 0\r\n },\r\n \"LicensedFeatures\": []\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"No description\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"folders\",\r\n \"Description\": \"Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder.\",\r\n \"Usage\": \"uip or folders [options] [command]\",\r\n \"Arguments\": [],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Subcommands\": [\r\n {\r\n \"Name\": \"list [options]\",\r\n \"Description\": \"List folders the current user has access to. Use --all to list all folders in the tenant (supports filtering and sorting).\"\r\n },\r\n {\r\n \"Name\": \"create [options] \",\r\n \"Description\": \"Create a new folder. Folders scope processes, jobs, and resources with separate permissions. Use --parent to nest inside an existing folder.\"\r\n },\r\n {\r\n \"Name\": \"get [options] \",\r\n \"Description\": \"Get folder details by key (GUID) or path. Returns display name, key (GUID), path, type, and permissions. Use 'folders list' to find folder keys.\"\r\n },\r\n {\r\n \"Name\": \"delete [options] \",\r\n \"Description\": \"Delete a folder by key (GUID) or path. Fails if the folder contains entities or user associations. Move or remove contents before deleting.\"\r\n },\r\n {\r\n \"Name\": \"move [options] \",\r\n \"Description\": \"Move a folder to a new parent in the folder hierarchy. Provide the target parent folder path or key. Use --root to move to the top level.\"\r\n },\r\n {\r\n \"Name\": \"edit [options] \",\r\n \"Description\": \"Edit folder properties (display name, description). Use 'folders get' to see current values before editing.\"\r\n },\r\n {\r\n \"Name\": \"runtimes [options] \",\r\n \"Description\": \"List runtime allocations for a folder. Shows how many runtime slots of each type (Unattended, Attended, NonProduction, etc.) are total, connected, and available in the specified folder. Useful for checking capacity before starting jobs.\"\r\n },\r\n {\r\n \"Name\": \"help [command]\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"" + }, + { + "argv": [ + "or", + "folders", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderGet\",\r\n \"Data\": {\r\n \"Name\": \"Shared\",\r\n \"ID\": 1500287,\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Description\": \"No description\",\r\n \"Path\": \"Shared\",\r\n \"ParentID\": \"Root\",\r\n \"FolderType\": \"Standard\",\r\n \"IsPersonal\": \"No\",\r\n \"ProvisionType\": \"Automatic\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"RuntimeType\": \"StudioPro\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"EntryPointPath\": \"Google.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"ErrorCode\": \"Robot\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 64312253\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs get d2c90d73", + "min": 1, + "description": "Agent fetches the faulted job to confirm AutopilotForRobots.Enabled, HealingEnabled, RuntimeType, State=Faulted (playbook step 2)." + }, + { + "pattern": "or jobs logs d2c90d73", + "min": 1, + "description": "Agent fetches Error-level robot logs to find the 'No available license / Agentic units' line (playbook step 1, the deterministic signal)." + }, + { + "pattern": "or licenses info", + "min": 1, + "description": "Agent reads tenant license to discriminate the playbook cause (#1 missing entitlement vs #7 exhausted) — Allowed.AgentService=0, LicensedFeatures=[] confirms #1 (playbook step 5)." + }, + { + "pattern": "or jobs healing-data d2c90d73", + "min": 0, + "description": "Optional: corroborates with empty-ZIP marker (playbook step 3). Agent may skip this when the Error log + licenses info already deterministically identify cause #1." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/task.yaml index 7b8155e972..048c8a4519 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-invalid-license/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/README.md new file mode 100644 index 0000000000..7f5b06e332 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/README.md @@ -0,0 +1,42 @@ +# Healing Agent No License — Faithful Replay + +This scenario replays a real UiPath diagnostic investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +A Faulted Orchestrator job for process **ERN** in folder **Shared** (job key `cedeff5a-ba9b-4ff5-80c2-b79b5662f59b`) failed with `UiPath.UIAutomationNext.Exceptions.NodeNotFoundException` at the **Click 'Simt că am noroc'** activity in `Google.xaml` — an authoring-time selector typo (`aria-label='Simt că am noroccccccccccc'` with trailing `cccccccccc` vs the live `aria-label='Simt că am noroc'` at 94% match). + +`AutopilotForRobots.Enabled=true` and `HealingEnabled=true` on the job, so Healing Agent was invoked — but it refused to run. The robot Error log at `2026-05-18T12:49:12.651Z` contains the deterministic line *"'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery."*, `uip or jobs healing-data` returned a 22-byte empty ZIP, and `uip or licenses info` confirmed `Allowed.AgentService=0` with `LicensedFeatures=[]` despite `SubscriptionPlan='ENTERPRISE'`. + +The match is **cause #1** in `healing-agent-no-license.md` — *No Healing Agent entitlement on the tenant*. RuntimeType=`StudioPro` (Non-Test set) means the requested pool is regular **Heals** (operation code `HealingAgent`), not Test Heals. Fix: acquire the Healing Agent Add-On for the tenant, or move to Flex Advanced / Unified Enterprise. + +**Difference vs `healing-agent-invalid-license`:** the initial prompt here ("Why my last job from folder Shared has failed?") does NOT provide the job key, so the agent must list Faulted jobs in folder Shared and pick the most recent one before fetching get/logs/healing-data. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-diagnostics` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name healing-agent-no-license --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/0182fb9f82.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/0182fb9f82.json deleted file mode 100644 index bf65fcd5e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/0182fb9f82.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/06e5cd9129.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/06e5cd9129.json deleted file mode 100644 index 142758a9fe..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/06e5cd9129.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-05-18T12:49:13.443Z", - "JobId": 64476250 - }, - { - "State": "Running", - "Timestamp": "2026-05-18T12:48:38.773Z", - "JobId": 64476250 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/22f7b79fc4.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/22f7b79fc4.json deleted file mode 100644 index 1211636d80..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/22f7b79fc4.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHealingData", - "Data": { - "JobKey": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "FileName": "", - "Size": 22, - "ExportId": 11660 - } -} -OK -22 \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/259aa2aff6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/259aa2aff6.json deleted file mode 100644 index 07f8bce47f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/259aa2aff6.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "LicenseInfo", - "Data": { - "IsExpired": false, - "SubscriptionPlan": "ENTERPRISE", - "Allowed": { - "StudioX": 0, - "TestAutomation": 1, - "Headless": 0, - "PerformanceTest": 0, - "Attended": 0, - "AgentService": 0, - "Unattended": 1, - "Development": 0, - "BusinessRule": 0, - "CaseManagement": 0, - "ProcessOrchestration": 0, - "AppTest": 0, - "Flow": 0, - "Hosting": 0, - "StudioPro": 0, - "NonProduction": 0, - "Agent": 0 - }, - "Used": { - "StudioX": 0, - "TestAutomation": 0, - "Headless": 0, - "PerformanceTest": 0, - "Attended": 0, - "AgentService": 0, - "Unattended": 1, - "Development": 0, - "BusinessRule": 0, - "CaseManagement": 0, - "ProcessOrchestration": 0, - "AppTest": 0, - "Flow": 0, - "Hosting": 0, - "StudioPro": 0, - "NonProduction": 0, - "Agent": 0 - }, - "LicensedFeatures": [] - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/README.md deleted file mode 100644 index 2d8987160f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# Healing Agent No License — Faithful Replay - -This scenario replays a real UiPath diagnostic investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -A Faulted Orchestrator job for process **ERN** in folder **Shared** (job key `cedeff5a-ba9b-4ff5-80c2-b79b5662f59b`) failed with `UiPath.UIAutomationNext.Exceptions.NodeNotFoundException` at the **Click 'Simt că am noroc'** activity in `Google.xaml` — an authoring-time selector typo (`aria-label='Simt că am noroccccccccccc'` with trailing `cccccccccc` vs the live `aria-label='Simt că am noroc'` at 94% match). - -`AutopilotForRobots.Enabled=true` and `HealingEnabled=true` on the job, so Healing Agent was invoked — but it refused to run. The robot Error log at `2026-05-18T12:49:12.651Z` contains the deterministic line *"'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery."*, `uip or jobs healing-data` returned a 22-byte empty ZIP, and `uip or licenses info` confirmed `Allowed.AgentService=0` with `LicensedFeatures=[]` despite `SubscriptionPlan='ENTERPRISE'`. - -The match is **cause #1** in `healing-agent-no-license.md` — *No Healing Agent entitlement on the tenant*. RuntimeType=`StudioPro` (Non-Test set) means the requested pool is regular **Heals** (operation code `HealingAgent`), not Test Heals. Fix: acquire the Healing Agent Add-On for the tenant, or move to Flex Advanced / Unified Enterprise. - -**Difference vs `healing-agent-invalid-license`:** the initial prompt here ("Why my last job from folder Shared has failed?") does NOT provide the job key, so the agent must list Faulted jobs in folder Shared and pick the most recent one before fetching get/logs/healing-data. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-diagnostics` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name healing-agent-no-license --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/aee7e0da81.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/aee7e0da81.json deleted file mode 100644 index 9f9e851f34..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/aee7e0da81.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-18T12:49:13.058Z", - "Message": "Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-18T12:49:12.651Z", - "Message": "'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/c00cbfea09.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/c00cbfea09.json deleted file mode 100644 index 5cece690a7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/c00cbfea09.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "Info": "Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\n\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-18T12:48:38.730Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20182479, - "EntryPointPath": "Google.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-18T12:49:13.423Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "cedeff5aba9b4ff580c2b79b5662f59b", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 64476250 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/manifest.json deleted file mode 100644 index 231419da8f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted --output json", - "file": "0182fb9f82.json" - }, - { - "match": "or jobs get cedeff5a-ba9b-4ff5-80c2-b79b5662f59b --output json", - "file": "c00cbfea09.json" - }, - { - "match": "or jobs logs cedeff5a-ba9b-4ff5-80c2-b79b5662f59b --level Error --output json", - "file": "aee7e0da81.json" - }, - { - "match": "or jobs history cedeff5a-ba9b-4ff5-80c2-b79b5662f59b --output json", - "file": "06e5cd9129.json" - }, - { - "match": "or jobs healing-data cedeff5a-ba9b-4ff5-80c2-b79b5662f59b -o ''", - "file": "22f7b79fc4.json" - }, - { - "match": "or licenses info --output json", - "file": "259aa2aff6.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/uip-fixture.json new file mode 100644 index 0000000000..632135197b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/data/uip-fixture.json @@ -0,0 +1,103 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-18T12:49:13.058Z\",\r\n \"Message\": \"Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-18T12:49:12.651Z\",\r\n \"Message\": \"'Click 'Simt că am noroc'' activity recovery failed. No available license / Agentic units to perform healing analysis and recovery.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data", + "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", + "-o", + "" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHealingData\",\r\n \"Data\": {\r\n \"JobKey\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"FileName\": \"\",\r\n \"Size\": 22,\r\n \"ExportId\": 11660\r\n }\r\n}\r\nOK\r\r\n22" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"Info\": \"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20182479,\r\n \"EntryPointPath\": \"Google.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"cedeff5aba9b4ff580c2b79b5662f59b\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 64476250\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-18T12:49:13.443Z\",\r\n \"JobId\": 64476250\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-18T12:48:38.773Z\",\r\n \"JobId\": 64476250\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "licenses", + "info", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LicenseInfo\",\r\n \"Data\": {\r\n \"IsExpired\": false,\r\n \"SubscriptionPlan\": \"ENTERPRISE\",\r\n \"Allowed\": {\r\n \"StudioX\": 0,\r\n \"TestAutomation\": 1,\r\n \"Headless\": 0,\r\n \"PerformanceTest\": 0,\r\n \"Attended\": 0,\r\n \"AgentService\": 0,\r\n \"Unattended\": 1,\r\n \"Development\": 0,\r\n \"BusinessRule\": 0,\r\n \"CaseManagement\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"AppTest\": 0,\r\n \"Flow\": 0,\r\n \"Hosting\": 0,\r\n \"StudioPro\": 0,\r\n \"NonProduction\": 0,\r\n \"Agent\": 0\r\n },\r\n \"Used\": {\r\n \"StudioX\": 0,\r\n \"TestAutomation\": 0,\r\n \"Headless\": 0,\r\n \"PerformanceTest\": 0,\r\n \"Attended\": 0,\r\n \"AgentService\": 0,\r\n \"Unattended\": 1,\r\n \"Development\": 0,\r\n \"BusinessRule\": 0,\r\n \"CaseManagement\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"AppTest\": 0,\r\n \"Flow\": 0,\r\n \"Hosting\": 0,\r\n \"StudioPro\": 0,\r\n \"NonProduction\": 0,\r\n \"Agent\": 0\r\n },\r\n \"LicensedFeatures\": []\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/task.yaml index 58da2dd238..f6d908ab34 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-no-license/task.yaml @@ -23,22 +23,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/README.md new file mode 100644 index 0000000000..1833e35bdb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/README.md @@ -0,0 +1,36 @@ +# Healing Agent Recommendation Only — Faithful Replay + +This scenario replays a real UiPath diagnostic investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Warning:** Healing Agent was running in recommendation-only mode (OrchestratorEnableHeal=false) — the recovered selector was inferred from the UI tree after the failure but was not validated at runtime. There is no guarantee it will work. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-diagnostics` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name healing-agent-recommendation-only --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/0f9cc6abcf.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/0f9cc6abcf.json deleted file mode 100644 index e44954ebfa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/0f9cc6abcf.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "GetErrors", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/171e2db2f1.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/171e2db2f1.json deleted file mode 100644 index 5452221249..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/171e2db2f1.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "CreationTime": "2026-05-12T08:13:00.720Z", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "HostMachineName": "MOCK-HOST", - "Reason": "NodeNotFoundException" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/242d4a5b96.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/242d4a5b96.json deleted file mode 100644 index d8a9b4f4bb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/242d4a5b96.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "Info": "Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\n\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-12T08:13:00.720Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53409, - "EntryPointPath": "Google.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T08:14:11.487Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d5fed611e740406ca1b03c6de3371f17", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "AutoHealStatus": "IssueDetected", - "Id": 4014085 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/591a39c495.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/591a39c495.json deleted file mode 100644 index 5c0c931662..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/591a39c495.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-12T08:14:10.608Z", - "Message": "ERN execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T08:14:10.283Z", - "Message": "Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] )\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException\r\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T08:14:08.818Z", - "Message": "Healing agent could not recover the activity. Self-healing is disabled." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T08:13:35.580Z", - "Message": "Healing agent: starting recovery analysis." - }, - { - "Level": "Warn", - "TimeStamp": "2026-05-12T08:13:35.580Z", - "Message": "'Click 'Simt că am noroc'' execution failed with exception NodeNotFoundException." - }, - { - "Level": "Warn", - "TimeStamp": "2026-05-12T08:13:32.408Z", - "Message": "Click 'Simt că am noroc': [Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[94%] \n[82%] \n[69%] \n[57%] " - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T08:13:00.952Z", - "Message": "Healing agent analysis is enabled but recovery is disabled for current job." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T08:13:00.379Z", - "Message": "ERN execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/README.md deleted file mode 100644 index d87f221774..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Healing Agent Recommendation Only — Faithful Replay - -This scenario replays a real UiPath diagnostic investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Warning:** Healing Agent was running in recommendation-only mode (OrchestratorEnableHeal=false) — the recovered selector was inferred from the UI tree after the failure but was not validated at runtime. There is no guarantee it will work. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-diagnostics` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name healing-agent-recommendation-only --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/b914c17fd0.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/b914c17fd0.json deleted file mode 100644 index c5bb62de2e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/b914c17fd0.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-39e2-d2e24e1a2cc2", - "TraceId": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "ParentId": null, - "Name": "Robot Job d5fed611-e740-406c-a1b0-3c6de3371f17", - "StartTime": "2026-05-12T08:13:00.3726258Z", - "EndTime": "2026-05-12T08:14:10.2930246Z", - "Attributes": "{\"Exception\":\"UiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "b16092a1-0c68-436c-bf3a-147ab53b6762", - "JobKey": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-05-12T08:14:11.3129343Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/c63026ca6f.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/c63026ca6f.json deleted file mode 100644 index e81c0afe2b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/c63026ca6f.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHealingData", - "Message": "Healing recovery data zip for this job \u00e2\u20ac\u201d see project .uia/ artifacts. RecoverySuccessful=false, mode=recommendation-only.", - "Data": { - "JobKey": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "HealingRecoveryAvailable": true, - "RecoverySuccessful": false, - "AnalysisMode": "RecommendationOnly", - "Note": "Recovery zip materialized into project at .uia/improve-NClick_1/. Use those files directly." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/e0c921e32c.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/e0c921e32c.json deleted file mode 100644 index 4eb8a6aa1c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/e0c921e32c.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d5fed611-e740-406c-a1b0-3c6de3371f17", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:13:00.720Z", - "EndTime": "2026-05-12T08:14:11.487Z", - "CreationTime": "2026-05-12T08:13:00.720Z" - }, - { - "Key": "8d7828fe-941f-4e53-8f9c-beea54a05346", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T07:22:23.770Z", - "EndTime": "2026-05-12T07:22:24.613Z", - "CreationTime": "2026-05-12T07:22:23.770Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/manifest.json deleted file mode 100644 index 5f01ca0d5d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/m/r/manifest.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of a UIA selector-failure investigation. Triage rules captured verbatim; healing-data/history/get-errors stubbed so the agent can explore beyond the recorded path without aborting.", - "rules": [ - { - "match": "or folders list --output json", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted --output json", - "file": "e0c921e32c.json" - }, - { - "match": "or jobs list", - "file": "e0c921e32c.json", - "_doc": "Catch-all: any `or jobs list` (regardless of filters) returns the recorded faulted-jobs list. Lets the agent succeed even if it omits --state Faulted." - }, - { - "match": "or jobs get d5fed611-e740-406c-a1b0-3c6de3371f17", - "file": "242d4a5b96.json" - }, - { - "match": "or jobs logs d5fed611-e740-406c-a1b0-3c6de3371f17", - "file": "591a39c495.json", - "_doc": "Recorded error-level logs that include the NodeNotFoundException + the Healing Agent recommendation-only message." - }, - { - "match": "or jobs history d5fed611-e740-406c-a1b0-3c6de3371f17", - "file": "171e2db2f1.json", - "_doc": "Stubbed: minimal job-history record showing the single Faulted transition. The original session did not redirect this command." - }, - { - "match": "or jobs healing-data d5fed611-e740-406c-a1b0-3c6de3371f17", - "file": "c63026ca6f.json", - "_doc": "Stubbed: marker pointing the agent to the project `.uia/improve-NClick_1/` artifacts. The real HA recovery zip is materialized into the project snapshot." - }, - { - "match": "or jobs traces d5fed611-e740-406c-a1b0-3c6de3371f17", - "file": "b914c17fd0.json", - "_doc": "Recorded job traces." - }, - { - "match": "rpa validate", - "file": "0f9cc6abcf.json", - "_doc": "Clean validation result — used after the agent applies the selector fix and re-validates Google.xaml." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/uip-fixture.json new file mode 100644 index 0000000000..e817eee302 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/data/uip-fixture.json @@ -0,0 +1,111 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d5fed611-e740-406c-a1b0-3c6de3371f17" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"Info\": \"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53409,\r\n \"EntryPointPath\": \"Google.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d5fed611e740406ca1b03c6de3371f17\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"AutoHealStatus\": \"IssueDetected\",\r\n \"Id\": 4014085\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d5fed611-e740-406c-a1b0-3c6de3371f17" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T08:14:10.608Z\",\r\n \"Message\": \"ERN execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T08:14:10.283Z\",\r\n \"Message\": \"Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] )\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException\\r\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T08:14:08.818Z\",\r\n \"Message\": \"Healing agent could not recover the activity. Self-healing is disabled.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T08:13:35.580Z\",\r\n \"Message\": \"Healing agent: starting recovery analysis.\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-05-12T08:13:35.580Z\",\r\n \"Message\": \"'Click 'Simt că am noroc'' execution failed with exception NodeNotFoundException.\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-05-12T08:13:32.408Z\",\r\n \"Message\": \"Click 'Simt că am noroc': [Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[94%] \\n[82%] \\n[69%] \\n[57%] \"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T08:13:00.952Z\",\r\n \"Message\": \"Healing agent analysis is enabled but recovery is disabled for current job.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T08:13:00.379Z\",\r\n \"Message\": \"ERN execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "d5fed611-e740-406c-a1b0-3c6de3371f17" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"Reason\": \"NodeNotFoundException\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data", + "d5fed611-e740-406c-a1b0-3c6de3371f17" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHealingData\",\r\n \"Message\": \"Healing recovery data zip for this job \\u00e2\\u20ac\\u201d see project .uia/ artifacts. RecoverySuccessful=false, mode=recommendation-only.\",\r\n \"Data\": {\r\n \"JobKey\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"HealingRecoveryAvailable\": true,\r\n \"RecoverySuccessful\": false,\r\n \"AnalysisMode\": \"RecommendationOnly\",\r\n \"Note\": \"Recovery zip materialized into project at .uia/improve-NClick_1/. Use those files directly.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "d5fed611-e740-406c-a1b0-3c6de3371f17" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-39e2-d2e24e1a2cc2\",\r\n \"TraceId\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"StartTime\": \"2026-05-12T08:13:00.3726258Z\",\r\n \"EndTime\": \"2026-05-12T08:14:10.2930246Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\\\r\\\\n\\\\r\\\\nPossible solutions:\\\\r\\\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\\\r\\\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\\\r\\\\n • If needed, re-indicate the element as its properties might have changed\\\\r\\\\n • Use \\\\\\\"Check state\\\\\\\" activity to check the application state before executing the action\\\\r\\\\n • Increase the \\\\\\\"Delay before\\\\\\\" value to allow time to the application to render entirely and become responsive \\\\r\\\\n\\\\r\\\\n[Target]: Strict selector failure:\\\\r\\\\n\\\\n\\\\r\\\\nThe closest selector matches found are:\\\\r\\\\n[94%] \\\\n[82%] \\\\n[69%] \\\\n[57%] )\\\\r\\\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\\\r\\\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"b16092a1-0c68-436c-bf3a-147ab53b6762\",\r\n \"JobKey\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-05-12T08:14:11.3129343Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5fed611-e740-406c-a1b0-3c6de3371f17\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T08:13:00.720Z\",\r\n \"EndTime\": \"2026-05-12T08:14:11.487Z\",\r\n \"CreationTime\": \"2026-05-12T08:13:00.720Z\"\r\n },\r\n {\r\n \"Key\": \"8d7828fe-941f-4e53-8f9c-beea54a05346\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T07:22:23.770Z\",\r\n \"EndTime\": \"2026-05-12T07:22:24.613Z\",\r\n \"CreationTime\": \"2026-05-12T07:22:23.770Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "rpa", + "validate" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"GetErrors\",\r\n \"Data\": []\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/task.yaml index 562c5b0ff2..8276725a3f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-healing-agent-recommendation-only/task.yaml @@ -17,22 +17,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/README.md new file mode 100644 index 0000000000..73e4c8eed8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/README.md @@ -0,0 +1,33 @@ +# UIAutomationNext NApplicationCard — Could Not Generate View (Design-Time) + +Design-time (Studio) troubleshooting scenario for +`UiPath.UIAutomationNext.Activities.NApplicationCard` (the modern Use Application/Browser container). + +## What this scenario exercises + +Studio cannot render the `Use Application/Browser` card on the design surface — the Output panel shows +`Could not generate view for NApplicationCard` + `Object reference not set to an instance of an object`. +The agent must recognize this as a **Studio ↔ activity-package version skew + corrupted/stale cache** +problem (a design-time rendering fault, not a robot job fault, and not corrupt XAML), and prescribe +aligning the package version and clearing caches. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted project source: `Main.xaml` with a `Use Browser` `NApplicationCard`; `project.json` pins `UiPath.UIAutomation.Activities [24.10.12]` against an older `studioVersion 23.10.4.0`; a stale `obj/project.assets.json` restored against yet-older `22.10.7` — cache drift | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the Studio/package version skew + stale cache and the fix (align + `UiPath.UIAutomation.Activities` to the Studio/runtime version, clear `%LOCALAPPDATA%\UiPath\.cache` + and the project `.local`/`bin`/`obj`, reopen — do NOT hand-edit the `.xaml`). + +Playbook: `references/activity-packages/ui-automation/playbooks/napplicationcard-view-generation-failed.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/m/r/README.md deleted file mode 100644 index 6891e613f1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/m/r/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# UIAutomationNext NApplicationCard — Could Not Generate View (Design-Time) - -Design-time (Studio) troubleshooting scenario for -`UiPath.UIAutomationNext.Activities.NApplicationCard` (the modern Use Application/Browser container). - -## What this scenario exercises - -Studio cannot render the `Use Application/Browser` card on the design surface — the Output panel shows -`Could not generate view for NApplicationCard` + `Object reference not set to an instance of an object`. -The agent must recognize this as a **Studio ↔ activity-package version skew + corrupted/stale cache** -problem (a design-time rendering fault, not a robot job fault, and not corrupt XAML), and prescribe -aligning the package version and clearing caches. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted project source: `Main.xaml` with a `Use Browser` `NApplicationCard`; `project.json` pins `UiPath.UIAutomation.Activities [24.10.12]` against an older `studioVersion 23.10.4.0`; a stale `obj/project.assets.json` restored against yet-older `22.10.7` — cache drift | -| `data/m/r/manifest.json` | `docsai ask` passthrough + permissive empty `unmocked_default` — no job/trace exists for a design-time error | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the Studio/package version skew + stale cache and the fix (align - `UiPath.UIAutomation.Activities` to the Studio/runtime version, clear `%LOCALAPPDATA%\UiPath\.cache` - and the project `.local`/`bin`/`obj`, reopen — do NOT hand-edit the `.xaml`). - -Playbook: `references/activity-packages/ui-automation/playbooks/napplicationcard-view-generation-failed.md`. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/m/r/manifest.json deleted file mode 100644 index 8d4a4a10b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/m/r/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. Only docsai is proxied to the real CLI; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/uip-fixture.json new file mode 100644 index 0000000000..bcb5501faa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/data/uip-fixture.json @@ -0,0 +1,8 @@ +{ + "version": 1, + "responses": [], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/task.yaml index 69e98cbccc..83b11b70bb 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-napplicationcard-view-generation/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/README.md new file mode 100644 index 0000000000..523ee29330 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/README.md @@ -0,0 +1,36 @@ +# No Healing Agent — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** Authoring-time corruption of the click target's `aria-label` selector attribute — the value `'Simt că am noroccccccccccc'` (ten extra `c` characters appended) does not match any real element on the Google homepage, so the Click activity cannot resolve its target and faults. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name no-healing-agent --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/0901a66f7e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/0901a66f7e.json deleted file mode 100644 index 21c116ed16..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/0901a66f7e.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-05-25T10:39:07.183Z", - "JobId": 65108299 - }, - { - "State": "Running", - "Timestamp": "2026-05-25T10:38:34.507Z", - "JobId": 65108299 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/1ca7b05e92.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/1ca7b05e92.json deleted file mode 100644 index 431d12734a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/1ca7b05e92.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "91d543d2-1b71-4233-a94a-a1389014c998", - "Info": "Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive\r\n\r\nin Google.xaml\r\n at NClick \"Click 'Simt c\u0103 am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-25T10:38:34.457Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20452595, - "EntryPointPath": "Google.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-25T10:39:07.113Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "91d543d21b714233a94aa1389014c998", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 65108299 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/739b120a16.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/739b120a16.json deleted file mode 100644 index de6f5ef8cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/739b120a16.json +++ /dev/null @@ -1,180 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 12, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/README.md deleted file mode 100644 index 3da0c93e30..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# No Healing Agent — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** Authoring-time corruption of the click target's `aria-label` selector attribute — the value `'Simt că am noroccccccccccc'` (ten extra `c` characters appended) does not match any real element on the Google homepage, so the Click activity cannot resolve its target and faults. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name no-healing-agent --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/d86a9b82a1.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/d86a9b82a1.json deleted file mode 100644 index 822f9e51ed..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/d86a9b82a1.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-25T10:39:06.234Z", - "Message": "Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n • Ensure application is opened and the UI element is visible on the screen at execution time\r\n • Edit the Target of the UI activity and use Validation to debug the issue.\r\n • If needed, re-indicate the element as its properties might have changed\r\n • Use \"Check state\" activity to check the application state before executing the action\r\n • Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive\r\n\r\nin Google.xaml\r\n at NClick \"Click 'Simt că am noroc'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"Google\"\r\n at Google \"Google\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/manifest.json deleted file mode 100644 index e89f3e6722..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted --output json", - "file": "739b120a16.json" - }, - { - "match": "or jobs get 91d543d2-1b71-4233-a94a-a1389014c998 --output json", - "file": "1ca7b05e92.json" - }, - { - "match": "or jobs history 91d543d2-1b71-4233-a94a-a1389014c998 --output json", - "file": "0901a66f7e.json" - }, - { - "match": "or jobs logs 91d543d2-1b71-4233-a94a-a1389014c998 --level Error --output json", - "file": "d86a9b82a1.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai query strings vary between runs; proxy to the real `uip docsai ask` on the host. Dispatcher caches each unique query under responses/_cache/ for in-run reuse." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/uip-fixture.json new file mode 100644 index 0000000000..f9a709057e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 12,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "91d543d2-1b71-4233-a94a-a1389014c998", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-25T10:39:06.234Z\",\r\n \"Message\": \"Click 'Simt că am noroc': Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive\\r\\n\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt că am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "91d543d2-1b71-4233-a94a-a1389014c998", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"Info\": \"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive\\r\\n\\r\\nin Google.xaml\\r\\n at NClick \\\"Click 'Simt c\\u0103 am noroc'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"Google\\\"\\r\\n at Google \\\"Google\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive\\r\\n at UiPath.UIAutomationNext.Activities.TargetCommonLogic.GetSearchResultAsync(IRuntimeContext runtimeContext, ITargetAnchorable target, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.SearchAndSetTargetAsync(IRuntimeContext context, ClickData clickData, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20452595,\r\n \"EntryPointPath\": \"Google.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"91d543d21b714233a94aa1389014c998\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 65108299\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "91d543d2-1b71-4233-a94a-a1389014c998", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-25T10:39:07.183Z\",\r\n \"JobId\": 65108299\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-25T10:38:34.507Z\",\r\n \"JobId\": 65108299\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/task.yaml index 9d5c52779a..6fcb983c6d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-no-healing-agent/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/README.md new file mode 100644 index 0000000000..682028a34b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/README.md @@ -0,0 +1,36 @@ +# UIA Node Not Found — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Post-presentation actions** + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name uia-node-not-found --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/0ad9b64589.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/0ad9b64589.json deleted file mode 100644 index 0268ddbe0d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/0ad9b64589.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "01b78c23-37a7-4725-9ee2-9aac48234088", - "text": "The provided UiPath documentation does not mention the Google Doodles archive page, the `glue-drawer` element, or its interaction behavior. \n\nBased only on these documents, there is no information about:\n- Whether `glue-drawer` exists on that page.\n- Whether it opens automatically or requires a user click (or other interaction) to open.\n\nSo, from the available docs, we cannot determine if the `glue-drawer` on the Google Doodles archive page requires a user click to open.\n", - "sources": [ - "https://docs-staging.uipath.com/automation-cloud/automation-cloud/latest/admin-guide/exploring-the-user-interface", - "https://docs-staging.uipath.com/studio/standalone/latest/user-guide/the-user-interface" - ], - "disclaimers": [], - "followups": [ - "Are you trying to automate interactions with the Google Doodles page using UiPath Studio?", - "Do you need guidance on how UiPath selectors or UI automation handle custom web components like drawers?", - "Would you like help inspecting the Google Doodles page structure using UiPath UI Explorer or browser dev tools?" - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/1175724510.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/1175724510.json deleted file mode 100644 index 5df8de0021..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/1175724510.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "2cab01fc-65d2-4469-b98f-47b7c0dff85e", - "text": "The behavior of `doodles.google.com` (or the Google Doodles homepage) is not covered in the provided UiPath documentation.\n\nBased on general web behavior (not UiPath\u2011specific docs):\n\n- The **main Doodles homepage** typically shows:\n - The **current/featured doodle** for your region/date.\n - Plus navigation or tiles to browse **recent and past doodles** (an archive-style list).\n\n- The **archive views** (for example, URLs with date filters, categories, or search parameters) usually render:\n - A **list/grid of multiple doodles** matching that filter (e.g., all doodles from a given year or for a specific country/event).\n - Clicking a specific doodle in the archive opens a **detail page** for that single doodle.\n\nSo:\n- A generic homepage URL tends to emphasize **one main doodle plus some list navigation**.\n- Archive\u2011style URLs tend to show a **list/grid of doodles**.\n- Individual doodle URLs show **one doodle** with its description.\n\nThere is no UiPath document in the provided set that describes Google Doodles URL behavior, so none can be cited as a source for this answer.\n\n```json\n\n```\n\n```json```", - "sources": [], - "disclaimers": [], - "followups": [ - "Can you explain how to automate interaction with a website like Google Doodles using UiPath Studio or Studio Web?", - "How can I configure network or firewall rules so UiPath robots can access external sites like Google?", - "What UiPath documentation URLs do I need to allow so I can read docs from a restricted network?" - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/28c1388c3a.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/28c1388c3a.json deleted file mode 100644 index 90d86fc576..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/28c1388c3a.json +++ /dev/null @@ -1 +0,0 @@ --rw-r--r-- 1 UIPATH+user1 4096 129 May 20 11:29 .local/investigations/raw/triage-job-traces.json \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/35afc8e2f4.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/35afc8e2f4.json deleted file mode 100644 index 026f088504..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/35afc8e2f4.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "5878d901-7741-48c7-84e1-1e48480a638e", - "text": "To handle faulted jobs in Orchestrator, use the built\u2011in job management options and, if enabled, recordings to analyze failures. Retries themselves are not described in the provided docs, so I\u2019ll focus on what is documented.\n\n**1. Understand and identify faulted jobs**\n\n- A job is **Faulted** if it failed to start or the process threw an unhandled error during execution. Faulted jobs **must be restarted manually**. \n \u2013 See [Job States](https://docs-staging.uipath.com/orchestrator/automation-suite/2.2510/user-guide/job-states).\n\n**2. Inspect why the job faulted**\n\n- In **Jobs**:\n - Click **Details** to open the **Job Details** window and see why the job faulted (Robot, process, environment, start/end times, input/output, etc.). \n - Click **View Logs** (More Actions > **View Logs**) to open the **Logs** page for that job. \n \u2013 From [Managing Jobs](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs) and [Job States](https://docs-staging.uipath.com/orchestrator/automation-suite/2.2510/user-guide/job-states).\n\n- For unattended faulted jobs with **Enable Recording** turned on:\n - Use **More Options > Download Recording** to get the execution media and review the last moments before failure. \n \u2013 See [About Jobs \u2013 Recording](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/about-jobs) and [Job States](https://docs-staging.uipath.com/orchestrator/automation-suite/2.2510/user-guide/job-states).\n\n**3. Restart faulted jobs**\n\n- From the **Jobs** list:\n - Click **More Actions > Restart** on a job in **Faulted**, **Stopped**, or **Successful** state. \n - Adjust settings in the **Start Job** window if needed, then click **Start**. \n \u2013 From [Managing Jobs](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs).\n\n**4. Retry policies**\n\n- The provided documents do **not** describe configuring automatic retry policies for faulted jobs or processes. Those settings (e.g., queue item retries, process\u2011level retry logic) are outside the scope of the included pages.\n\n---\n", - "sources": [ - "https://docs-staging.uipath.com/orchestrator/automation-suite/2.2510/user-guide/job-states", - "https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs", - "https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/about-jobs" - ], - "disclaimers": [ - "Installation and administration instructions, if incorrectly implemented, can result in system disruptions or data loss.\nVerify if your scenario aligns with the documentation sources linked for this answer. We recommend creating a backup before making any changes." - ], - "followups": [ - "How can I design retries in Studio workflows when a job or activity fails?", - "How do I configure queue-based retry policies for failed queue items in Orchestrator?", - "Can you help me interpret a specific faulted job\u2019s logs or error message?" - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/4c614a390b.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/4c614a390b.json deleted file mode 100644 index 1d98467671..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/4c614a390b.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "be879fc8-5dfa-4cc8-88fa-7b061e7256f2", - "text": "The provided documents do not cover selector failures related to:\n\n- Regional/locale differences \n- Browser language causing different page content/DOM \n- `NodeNotFoundException` specifically in that context \n\nFrom a support perspective, when selectors fail due to locale or content differences, the usual approaches are:\n\n- Ensure the UI structure (DOM/window hierarchy) is the same across languages/regions. If not, use:\n - More robust, partial, or wildcard selectors\n - Anchor-based activities\n - Image-based or text-based targeting as a fallback\n- Avoid language\u2011dependent attributes (e.g., inner text that changes with locale) in selectors.\n- If using web apps, confirm that the same URL and layout are loaded regardless of browser language, or build separate selectors per locale and branch logic.\n- Reproduce the issue with UiExplorer to compare working vs failing selectors and adjust attributes accordingly.\n\nSince this scenario is not documented in the supplied articles, if the problem persists, capture:\n- Exact error (`NodeNotFoundException` stack trace)\n- Screenshots of the UI in each locale\n- Selector dumps from UiExplorer \nand open a UiPath support ticket.\n", - "sources": [], - "disclaimers": [], - "followups": [ - "Can you share the exact selector and the full NodeNotFoundException message you are getting?", - "Is the application you are automating a web app or desktop app, and which browser or technology is it using?", - "Do you see visible layout or text changes between the different regional or language settings?" - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/5baaaec208.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/5baaaec208.json deleted file mode 100644 index 0058bafa5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/5baaaec208.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "95eecd44d4e6203d", - "TraceId": "899a0c10d81b44ea9d29cecc00fd8976", - "ParentId": null, - "Name": "Robot Job 899a0c10-d81b-44ea-9d29-cecc00fd8976", - "StartTime": "2026-05-19T10:03:21.5009923Z", - "EndTime": "2026-05-19T10:04:04.1731647Z", - "Attributes": "{\"Exception\":\"UiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[39%] \\n[38%] \\n[38%] \\n[38%] \\n[37%] \\n[37%] \\n[36%] \\n[36%] \\n[35%] \\n[34%] )\\r\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "c81daadb-3ad3-4681-9fd8-627e57742f4b", - "TenantId": "3ea4784c-f5f9-4c9f-9cd9-a81c64569726", - "ExpiryTimeUtc": null, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "2c983759-2712-4740-b643-df8336716510", - "JobKey": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-05-19T10:04:04.7969287Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/6978cca2fc.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/6978cca2fc.json deleted file mode 100644 index 58292af674..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/6978cca2fc.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-05-19T10:04:05.013Z", - "JobId": 64565773 - }, - { - "State": "Running", - "Timestamp": "2026-05-19T10:03:21.630Z", - "JobId": 64565773 - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/6a483566b3.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/6a483566b3.json deleted file mode 100644 index 2f566696f2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/6a483566b3.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "a14788df-5ff8-4564-a212-1898fd5e8b61", - "text": "Use **Check App State** when the UI can be in *one of two states* (e.g., \u201cloading spinner visible\u201d vs \u201ccontent loaded\u201d), and you want to run different actions depending on which state you detect.\n\nUse **wait-for-element**\u2013type activities when you just need to ensure a specific element is ready before clicking.\n\nFrom the docs:\n\n- **[Check App State](https://docs-staging.uipath.com/activities/other/latest/ui-automation/n-check-state)** \n - Verifies if an element appears or disappears. \n - Can monitor an entire app for changes. \n - Lets you define:\n - \u201cTarget appears\u201d branch (element found)\n - \u201cTarget not appears\u201d branch (element not found) \n - Can be used inside or outside **Use Application/Browser** and as a condition for **Retry Scope**.\n\n- **Classic wait/exists activities** (from *UI Automation \u2013 UI Synchronization*): \n - [Element Exists](https://docs-staging.uipath.com/activities/docs/ui-element-exists), [Find Element](https://docs-staging.uipath.com/activities/docs/wait-ui-element-appear), [Wait Element Vanish](https://docs-staging.uipath.com/activities/docs/wait-ui-element-vanish), etc. \n - Designed to \u201cwait for the desired application state before other interactions\u201d.\n\n**For a dynamic content drawer not rendered yet:**\n- If you only care that the drawer\u2019s button/field is present before clicking \u2192 use **Find Element / Element Exists** as a wait step. \n- If you need logic for both \u201cdrawer loaded\u201d and \u201cdrawer not loaded / still loading\u201d \u2192 use **Check App State** and handle both branches.\n\n---\n", - "sources": [ - "https://docs-staging.uipath.com/activities/other/latest/ui-automation/n-check-state", - "https://docs-staging.uipath.com/studio/standalone/latest/user-guide/ui-automation" - ], - "disclaimers": [], - "followups": [ - "How do I configure a Check App State activity for a loading spinner vs loaded content?", - "Which properties should I set on Element Exists or Find Element to reliably wait for my drawer element?", - "Can you show a small workflow pattern combining Check App State with Retry Scope for unstable UIs?" - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/862b67f6c3.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/862b67f6c3.json deleted file mode 100644 index 6ea20aebc8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/862b67f6c3.json +++ /dev/null @@ -1,40 +0,0 @@ - uip or jobs -Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces. - Use --help with nested commands for more detail. - -┌─ Usage ──────────────────────────────────────────────────────────────────────┐ -│ uip or jobs [options] [command] │ -└──────────────────────────────────────────────────────────────────────────────┘ - -┌─ Commands ───────────────────────────────────────────────────────────────────┐ -│ get Get a job by its key (GUID). Returns state, input/output │ -│ arguments, timing, machine info, and error details if │ -│ faulted. If the job is in Pending state, also includes │ -│ diagnostic reasons (robot availability, capacity, etc.). No │ -│ folder context needed — the key is globally unique. Use 'jobs │ -│ list' to find job keys. │ -│ healing-data Download healing agent (Autopilot for Robots) recovery data │ -│ as a ZIP file. Contains screenshots, UI element data, and │ -│ recovery logs. Only available for jobs where the healing │ -│ agent was active. Triggers an async export, polls until │ -│ complete, then downloads the ZIP. │ -│ help display help for command │ -│ history Get state transition history for a job. Shows timestamps for │ -│ each state change (Pending → Running → Successful/Faulted). │ -│ Useful for understanding job lifecycle and identifying │ -│ delays. Folder context is resolved automatically from the job │ -│ key. │ -│ list List jobs. Returns job key, state, process name, timing, and │ -│ machine info. Use --state to filter by job state (Pending, │ -│ Running, Successful, Faulted, Stopped, Stopping, Suspended, │ -│ Resumed, Terminating). Provide --folder-path or --folder-key │ -│ to scope to a folder, or omit for cross-folder listing. │ -│ logs Get robot execution logs for a job. Filter by --level to │ -│ quickly find errors. Folder context is inferred from the job │ -│ key. Logs are returned newest-first. │ -│ restart Restart a faulted or stopped job. Creates a new job from the │ -│ same process. No folder context needed — resolved from job │ -│ key. │ -│ resume Resume a suspended job. Jobs become suspended when they use │ -│ the 'Suspend' activity (long-running workflows that wait for │ -│ external input). Optionally pass new --input-arguments to │ \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/README.md deleted file mode 100644 index e2f760925f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# UIA Node Not Found — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Post-presentation actions** - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name uia-node-not-found --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/be3a2553f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/be3a2553f5.json deleted file mode 100644 index 054bc6cd76..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/be3a2553f5.json +++ /dev/null @@ -1 +0,0 @@ --rw-r--r-- 1 UIPATH+user1 4096 3484 May 20 11:29 .local/investigations/raw/triage-job-logs-error.json \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/bfc60fcb97.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/bfc60fcb97.json deleted file mode 100644 index 3cdfbc6735..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/bfc60fcb97.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/e858500f53.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/e858500f53.json deleted file mode 100644 index 351858aeee..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/e858500f53.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "7efa7981-3ca6-4a2c-a5a8-2ad54e7844dd", - "text": "`AttachMode = ByInstance` in the API (`NAppAttachMode.ByInstance`) means:\n\n- Further UIA APIs will search **only in the indicated application instance**, including all parent/child windows (alerts, popups, etc.).\n- Other instances of the same app/browser are **excluded**. \n[Uipath Uiautomationnext Api Contracts Iuiautomationappservice Attach](https://docs-staging.uipath.com/activities/other/latest/ui-automation/uipath-uiautomationnext-api-contracts-iuiautomationappservice-attach)\n\nIf it \u201cattaches to the wrong page\u201d, typical causes (based on docs behavior):\n\n- The instance you attached to already has a different tab active.\n- Your later actions are not constrained to a specific tab element.\n\nRegarding navigation:\n\n- To change the page in the attached browser/tab, you should use a navigation activity such as **[Navigate Browser](https://docs-staging.uipath.com/activities/other/latest/ui-automation/n-navigate-browser)** inside **[Use Application/Browser](https://docs-staging.uipath.com/activities/other/latest/ui-automation/n-application-card)**.\n- `Navigate Browser` supports **Go Back / Go Forward / Refresh / Close Tab** (and **Go Home** in Windows\u2011Legacy).\n\nSo yes: if the attached instance is not already on the desired page, you need a navigation step (e.g. `Navigate Browser` or opening the correct URL in `Use Application/Browser`) to reach the correct page/tab.\n\n---\n", - "sources": [ - "https://docs-staging.uipath.com/activities/other/latest/ui-automation/uipath-uiautomationnext-api-contracts-iuiautomationappservice-attach", - "https://docs-staging.uipath.com/activities/other/latest/ui-automation/n-application-card", - "https://docs-staging.uipath.com/activities/other/latest/ui-automation/n-navigate-browser" - ], - "disclaimers": [], - "followups": [ - "How can I ensure Attach with ByInstance always targets a specific browser tab?", - "What is the difference between ByInstance, ByProcessName, and SingleWindow in NAppAttachMode?", - "How do I combine Use Application/Browser with Navigate Browser to control multiple tabs?" - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/f5ccb728f4.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/f5ccb728f4.json deleted file mode 100644 index 1bfb822f2c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/f5ccb728f4.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "Info": "Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[39%] \n[38%] \n[38%] \n[38%] \n[37%] \n[37%] \n[36%] \n[36%] \n[35%] \n[34%] )\r\n\r\nin DoodleClick.xaml\r\n at NClick \"Click 'Creating a Doodle'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"TO\"\r\n at TO \"TO\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\r\n\r\nPossible solutions:\r\n \u2022 Ensure application is opened and the UI element is visible on the screen at execution time\r\n \u2022 Edit the Target of the UI activity and use Validation to debug the issue.\r\n \u2022 If needed, re-indicate the element as its properties might have changed\r\n \u2022 Use \"Check state\" activity to check the application state before executing the action\r\n \u2022 Increase the \"Delay before\" value to allow time to the application to render entirely and become responsive \r\n\r\n[Target]: Strict selector failure:\r\n\n\r\nThe closest selector matches found are:\r\n[39%] \n[38%] \n[38%] \n[38%] \n[37%] \n[37%] \n[36%] \n[36%] \n[35%] \n[34%] )\r\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-19T10:03:21.587Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20182479, - "EntryPointPath": "DoodleClick.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T10:04:04.930Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "899a0c10d81b44ea9d29cecc00fd8976", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "AutoHealStatus": "IssueDetected", - "Id": 64565773 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/manifest.json deleted file mode 100644 index 21858538ca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py — manifest hand-tightened: match strings strip shell redirects and `-o` paths so substring matching tolerates whatever flags the agent passes; docsai consolidated to a single passthrough rule.", - "rules": [ - { - "match": "or folders list --output json", - "file": "bfc60fcb97.json" - }, - { - "match": "or jobs get 899a0c10-d81b-44ea-9d29-cecc00fd8976 --output json", - "file": "f5ccb728f4.json" - }, - { - "match": "or jobs logs 899a0c10-d81b-44ea-9d29-cecc00fd8976 --level Error", - "file": "be3a2553f5.json" - }, - { - "match": "or jobs history 899a0c10-d81b-44ea-9d29-cecc00fd8976", - "file": "6978cca2fc.json" - }, - { - "match": "or jobs --help", - "file": "862b67f6c3.json" - }, - { - "match": "traces spans get --job-key 899a0c10-d81b-44ea-9d29-cecc00fd8976", - "file": "28c1388c3a.json" - }, - { - "match": "traces spans get 899a0c10d81b44ea9d29cecc00fd8976", - "file": "5baaaec208.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..b40f2e1259 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/data/uip-fixture.json @@ -0,0 +1,91 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "899a0c10-d81b-44ea-9d29-cecc00fd8976", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "-rw-r--r-- 1 UIPATH+user1 4096 3484 May 20 11:29 .local/investigations/raw/triage-job-logs-error.json" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "899a0c10-d81b-44ea-9d29-cecc00fd8976" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "-rw-r--r-- 1 UIPATH+user1 4096 129 May 20 11:29 .local/investigations/raw/triage-job-traces.json" + }, + { + "argv": [ + "or", + "jobs", + "get", + "899a0c10-d81b-44ea-9d29-cecc00fd8976", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"Info\": \"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[39%] \\n[38%] \\n[38%] \\n[38%] \\n[37%] \\n[37%] \\n[36%] \\n[36%] \\n[35%] \\n[34%] )\\r\\n\\r\\nin DoodleClick.xaml\\r\\n at NClick \\\"Click 'Creating a Doodle'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"TO\\\"\\r\\n at TO \\\"TO\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\r\\n \\u2022 Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n \\u2022 Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[39%] \\n[38%] \\n[38%] \\n[38%] \\n[37%] \\n[37%] \\n[36%] \\n[36%] \\n[35%] \\n[34%] )\\r\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20182479,\r\n \"EntryPointPath\": \"DoodleClick.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"899a0c10d81b44ea9d29cecc00fd8976\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"AutoHealStatus\": \"IssueDetected\",\r\n \"Id\": 64565773\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "history", + "899a0c10-d81b-44ea-9d29-cecc00fd8976" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T10:04:05.013Z\",\r\n \"JobId\": 64565773\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T10:03:21.630Z\",\r\n \"JobId\": 64565773\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "traces", + "spans", + "get", + "899a0c10d81b44ea9d29cecc00fd8976" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"95eecd44d4e6203d\",\r\n \"TraceId\": \"899a0c10d81b44ea9d29cecc00fd8976\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"StartTime\": \"2026-05-19T10:03:21.5009923Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.1731647Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\\\r\\\\n\\\\r\\\\nPossible solutions:\\\\r\\\\n \\u2022 Ensure application is opened and the UI element is visible on the screen at execution time\\\\r\\\\n \\u2022 Edit the Target of the UI activity and use Validation to debug the issue.\\\\r\\\\n \\u2022 If needed, re-indicate the element as its properties might have changed\\\\r\\\\n \\u2022 Use \\\\\\\"Check state\\\\\\\" activity to check the application state before executing the action\\\\r\\\\n \\u2022 Increase the \\\\\\\"Delay before\\\\\\\" value to allow time to the application to render entirely and become responsive \\\\r\\\\n\\\\r\\\\n[Target]: Strict selector failure:\\\\r\\\\n\\\\n\\\\r\\\\nThe closest selector matches found are:\\\\r\\\\n[39%] \\\\n[38%] \\\\n[38%] \\\\n[38%] \\\\n[37%] \\\\n[37%] \\\\n[36%] \\\\n[36%] \\\\n[35%] \\\\n[34%] )\\\\r\\\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\\\r\\\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"c81daadb-3ad3-4681-9fd8-627e57742f4b\",\r\n \"TenantId\": \"3ea4784c-f5f9-4c9f-9cd9-a81c64569726\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"2c983759-2712-4740-b643-df8336716510\",\r\n \"JobKey\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-05-19T10:04:04.7969287Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": " uip or jobs \r\nManage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces.\r\n Use --help with nested commands for more detail. \r\n\r\n┌─ Usage ──────────────────────────────────────────────────────────────────────┐\r\n│ uip or jobs [options] [command] │\r\n└──────────────────────────────────────────────────────────────────────────────┘\r\n\r\n┌─ Commands ───────────────────────────────────────────────────────────────────┐\r\n│ get Get a job by its key (GUID). Returns state, input/output │\r\n│ arguments, timing, machine info, and error details if │\r\n│ faulted. If the job is in Pending state, also includes │\r\n│ diagnostic reasons (robot availability, capacity, etc.). No │\r\n│ folder context needed — the key is globally unique. Use 'jobs │\r\n│ list' to find job keys. │\r\n│ healing-data Download healing agent (Autopilot for Robots) recovery data │\r\n│ as a ZIP file. Contains screenshots, UI element data, and │\r\n│ recovery logs. Only available for jobs where the healing │\r\n│ agent was active. Triggers an async export, polls until │\r\n│ complete, then downloads the ZIP. │\r\n│ help display help for command │\r\n│ history Get state transition history for a job. Shows timestamps for │\r\n│ each state change (Pending → Running → Successful/Faulted). │\r\n│ Useful for understanding job lifecycle and identifying │\r\n│ delays. Folder context is resolved automatically from the job │\r\n│ key. │\r\n│ list List jobs. Returns job key, state, process name, timing, and │\r\n│ machine info. Use --state to filter by job state (Pending, │\r\n│ Running, Successful, Faulted, Stopped, Stopping, Suspended, │\r\n│ Resumed, Terminating). Provide --folder-path or --folder-key │\r\n│ to scope to a folder, or omit for cross-folder listing. │\r\n│ logs Get robot execution logs for a job. Filter by --level to │\r\n│ quickly find errors. Folder context is inferred from the job │\r\n│ key. Logs are returned newest-first. │\r\n│ restart Restart a faulted or stopped job. Creates a new job from the │\r\n│ same process. No folder context needed — resolved from job │\r\n│ key. │\r\n│ resume Resume a suspended job. Jobs become suspended when they use │\r\n│ the 'Suspend' activity (long-running workflows that wait for │\r\n│ external input). Optionally pass new --input-arguments to │" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/task.yaml index e1fcb6b7c2..d9cf6d98f7 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-node-not-found/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/README.md new file mode 100644 index 0000000000..b47cd6fff3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/README.md @@ -0,0 +1,36 @@ +# Uia Out Of Screen Bounds — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Action 1 — Apply universal fix: switch NClick_1 InteractionMode from HardwareEvents to ChromiumAPI** + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name uia-out-of-screen-bounds --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/528d77b927.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/528d77b927.json deleted file mode 100644 index c237408743..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/528d77b927.json +++ /dev/null @@ -1,204 +0,0 @@ -EXIT=0 -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 14, \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/57a3d67a79.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/57a3d67a79.json deleted file mode 100644 index 536b628587..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/57a3d67a79.json +++ /dev/null @@ -1,29 +0,0 @@ -EXIT=0 -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/609683359f.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/609683359f.json deleted file mode 100644 index d958095501..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/609683359f.json +++ /dev/null @@ -1,72 +0,0 @@ -EXIT=0 -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Pending", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/README.md deleted file mode 100644 index 2a36d37108..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Uia Out Of Screen Bounds — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Action 1 — Apply universal fix: switch NClick_1 InteractionMode from HardwareEvents to ChromiumAPI** - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name uia-out-of-screen-bounds --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/d5f25d4c78.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/d5f25d4c78.json deleted file mode 100644 index 2a7fb68890..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/d5f25d4c78.json +++ /dev/null @@ -1,46 +0,0 @@ -EXIT=0 -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "4c84401d-b8b8-4ea9-9d65-9d0f98d0971d", - "Info": "Cannot send input to UI element because it is outside of screen bounds.\r\n\r\nin EditorLink.xaml\r\n at NClick \"Click 'Learn More About Editor'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge DataTables Javascript table library\"\r\n at Sequence \"Kbd\"\r\n at Ofsb \"Ofsb\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.UiAutomationException: Cannot send input to UI element because it is outside of screen bounds.\r\n ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x800402bd\r\n at UiPath.UiNodeClass.Click(UiClickProperties clickProperties)\r\n at UiPath.NodeExtensions.ClickAsync(UiNode this, UiClickProperties clickProperties)\r\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass13_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\r\n --- End of inner exception stack trace ---\r\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.GetFriendly(COMException comException)\r\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.ThrowFriendly(COMException comException)\r\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\r\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass25_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\r\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\r\n at UiPath.UIAutomationNext.Services.UiInputService.ClickAsync(IClickData data, TimeoutToken token)\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_1.<b__1>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Services.ActivityActionService.ClickAsync(IClickData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Extensions.IActivityActionServiceExtensions.ExecuteAsync(IActivityActionService service, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.TargetBase.ActionAsync[T](IRuntimeDataCollectionContext runtimeContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.UiPath.UIAutomationNext.Activities.IVerifyExecutionActivity.ExecuteAsync(IRuntimeContext context, IActivityData data, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-25T17:56:43.687Z", - "ReleaseName": "ERN", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20452595, - "EntryPointPath": "EditorLink.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "newrobot", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-25T17:56:48.883Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "8c994220a19a4c6298686df7b61c97e3", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 65135970 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/manifest.json deleted file mode 100644 index 61a6f3b865..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Verbatim uip CLI responses captured from the source session. Matches are core uip subcommand substrings — the troubleshoot agent's shared.md redirect-to-raw pipeline wraps these in `> .local/.../raw.json && head -c N`, but the dispatcher's substring match still catches the wrapped form. docsai queries proxy to the live CLI per tests/tasks/uipath-troubleshoot/CLAUDE.md.", - "rules": [ - { - "match": "or folders list --output json", - "file": "57a3d67a79.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "528d77b927.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --output json", - "file": "609683359f.json" - }, - { - "match": "or jobs get 8c994220-a19a-4c62-9868-6df7b61c97e3", - "file": "d5f25d4c78.json" - }, - { - "match": "or jobs logs 8c994220-a19a-4c62-9868-6df7b61c97e3", - "file": "d5f25d4c78.json" - }, - { - "match": "traces spans get --job-key 8c994220-a19a-4c62-9868-6df7b61c97e3", - "file": "d5f25d4c78.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/uip-fixture.json new file mode 100644 index 0000000000..1d6b98212c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/data/uip-fixture.json @@ -0,0 +1,83 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 14," + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\"," + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "8c994220-a19a-4c62-9868-6df7b61c97e3" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"4c84401d-b8b8-4ea9-9d65-9d0f98d0971d\",\r\n \"Info\": \"Cannot send input to UI element because it is outside of screen bounds.\\r\\n\\r\\nin EditorLink.xaml\\r\\n at NClick \\\"Click 'Learn More About Editor'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge DataTables Javascript table library\\\"\\r\\n at Sequence \\\"Kbd\\\"\\r\\n at Ofsb \\\"Ofsb\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.UiAutomationException: Cannot send input to UI element because it is outside of screen bounds.\\r\\n ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x800402bd\\r\\n at UiPath.UiNodeClass.Click(UiClickProperties clickProperties)\\r\\n at UiPath.NodeExtensions.ClickAsync(UiNode this, UiClickProperties clickProperties)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.GetFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.ThrowFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass25_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.ClickAsync(IClickData data, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_1.<b__1>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.ClickAsync(IClickData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Extensions.IActivityActionServiceExtensions.ExecuteAsync(IActivityActionService service, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.TargetBase.ActionAsync[T](IRuntimeDataCollectionContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.UiPath.UIAutomationNext.Activities.IVerifyExecutionActivity.ExecuteAsync(IRuntimeContext context, IActivityData data, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20452595,\r\n \"EntryPointPath\": \"EditorLink.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8c994220a19a4c6298686df7b61c97e3\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 65135970\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "get", + "8c994220-a19a-4c62-9868-6df7b61c97e3" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"4c84401d-b8b8-4ea9-9d65-9d0f98d0971d\",\r\n \"Info\": \"Cannot send input to UI element because it is outside of screen bounds.\\r\\n\\r\\nin EditorLink.xaml\\r\\n at NClick \\\"Click 'Learn More About Editor'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge DataTables Javascript table library\\\"\\r\\n at Sequence \\\"Kbd\\\"\\r\\n at Ofsb \\\"Ofsb\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.UiAutomationException: Cannot send input to UI element because it is outside of screen bounds.\\r\\n ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x800402bd\\r\\n at UiPath.UiNodeClass.Click(UiClickProperties clickProperties)\\r\\n at UiPath.NodeExtensions.ClickAsync(UiNode this, UiClickProperties clickProperties)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.GetFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.ThrowFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass25_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.ClickAsync(IClickData data, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_1.<b__1>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.ClickAsync(IClickData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Extensions.IActivityActionServiceExtensions.ExecuteAsync(IActivityActionService service, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.TargetBase.ActionAsync[T](IRuntimeDataCollectionContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.UiPath.UIAutomationNext.Activities.IVerifyExecutionActivity.ExecuteAsync(IRuntimeContext context, IActivityData data, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20452595,\r\n \"EntryPointPath\": \"EditorLink.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8c994220a19a4c6298686df7b61c97e3\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 65135970\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "8c994220-a19a-4c62-9868-6df7b61c97e3" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"4c84401d-b8b8-4ea9-9d65-9d0f98d0971d\",\r\n \"Info\": \"Cannot send input to UI element because it is outside of screen bounds.\\r\\n\\r\\nin EditorLink.xaml\\r\\n at NClick \\\"Click 'Learn More About Editor'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge DataTables Javascript table library\\\"\\r\\n at Sequence \\\"Kbd\\\"\\r\\n at Ofsb \\\"Ofsb\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.UiAutomationException: Cannot send input to UI element because it is outside of screen bounds.\\r\\n ---> System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x800402bd\\r\\n at UiPath.UiNodeClass.Click(UiClickProperties clickProperties)\\r\\n at UiPath.NodeExtensions.ClickAsync(UiNode this, UiClickProperties clickProperties)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.GetFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Extensions.ExceptionExtensions.ThrowFriendly(COMException comException)\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.<>c__DisplayClass25_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.DriverServiceCore.WrapComAsync[T](Func`1 toExecute, Nullable`1 traceEventType)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.RetryAsync(Func`1 action, IUiNodeX node, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.UiInputService.ClickAsync(IClickData data, TimeoutToken token)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_1.<b__1>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.WithSystemInputLock(Boolean requiresInputLock, Func`1 action)\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.<>c__DisplayClass31_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Services.ActivityActionService.ClickAsync(IClickData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Extensions.IActivityActionServiceExtensions.ExecuteAsync(IActivityActionService service, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.TargetBase.ActionAsync[T](IRuntimeDataCollectionContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.UiPath.UIAutomationNext.Activities.IVerifyExecutionActivity.ExecuteAsync(IRuntimeContext context, IActivityData data, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20452595,\r\n \"EntryPointPath\": \"EditorLink.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"newrobot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"8c994220a19a4c6298686df7b61c97e3\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 65135970\r\n }\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "EXIT=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/task.yaml index 4cb427efc3..8887af2eed 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-out-of-screen-bounds/task.yaml @@ -26,21 +26,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/README.md new file mode 100644 index 0000000000..196bfde4ff --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/README.md @@ -0,0 +1,36 @@ +# Rpa Selector Healing Disabled — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause: Benefits Enrollment Portal UI Change — Employee Name Input Field Selector Broken** + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name rpa-selector-healing-disabled --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/269b3bd27b.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/269b3bd27b.json deleted file mode 100644 index ec4abd2e17..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/269b3bd27b.json +++ /dev/null @@ -1,127 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "update", - "Description": "Update process settings by key (GUID). No folder needed — resolves cross-folder. Only provided fields are updated (PATCH). Use 'processes get' to see current values.", - "Usage": "uip or processes update [options] ", - "Arguments": [ - { - "Name": "process-key", - "Description": "Process key (GUID)", - "Required": true - } - ], - "Options": [ - { - "Flags": "-t, --tenant ", - "Description": "Tenant (optional, defaults to value selected during auth)" - }, - { - "Flags": "-n, --name ", - "Description": "Process display name" - }, - { - "Flags": "-d, --description ", - "Description": "Process description" - }, - { - "Flags": "--entry-point ", - "Description": "Entry point workflow path (for multi-entry-point packages)" - }, - { - "Flags": "--input-arguments ", - "Description": "Default input arguments as JSON string" - }, - { - "Flags": "--job-priority ", - "Description": "Default job priority (Low, Normal, High)" - }, - { - "Flags": "--specific-priority ", - "Description": "Specific priority value (1-100). Mutually exclusive with --job-priority." - }, - { - "Flags": "--auto-update", - "Description": "Enable auto-update to latest package version" - }, - { - "Flags": "--no-auto-update", - "Description": "Disable auto-update to latest package version" - }, - { - "Flags": "--hidden-for-attended", - "Description": "Hide from attended robot users" - }, - { - "Flags": "--visible-for-attended", - "Description": "Show to attended robot users" - }, - { - "Flags": "--auto-create-triggers", - "Description": "Auto-create connected triggers on deploy" - }, - { - "Flags": "--no-auto-create-triggers", - "Description": "Disable auto-create connected triggers on deploy" - }, - { - "Flags": "--retention-period ", - "Description": "Job retention period in days (1–180)" - }, - { - "Flags": "--retention-action ", - "Description": "Retention action (Delete, Archive, None)" - }, - { - "Flags": "--retention-bucket ", - "Description": "Storage bucket ID for archived jobs (required when action is Archive)" - }, - { - "Flags": "--stale-retention-period ", - "Description": "Stale job retention period in days" - }, - { - "Flags": "--stale-retention-action ", - "Description": "Stale retention action (Delete, Archive, None)" - }, - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Examples": [ - { - "Description": "Change a process description", - "Command": "uip or processes update c3d4e5f6-0000-0000-0000-000000000001 --description \"Monthly invoice batch\"", - "Output": { - "Code": "ProcessUpdated", - "Data": { - "Key": "c3d4e5f6-0000-0000-0000-000000000001", - "Name": "InvoiceProcessing", - "Status": "Updated successfully" - } - } - } - ], - "Subcommands": [], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/3ed41069b3.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/3ed41069b3.json deleted file mode 100644 index 4c9498e7cf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/3ed41069b3.json +++ /dev/null @@ -1,143 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "jobs", - "Description": "Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces.", - "Usage": "uip or jobs [options] [command]", - "Arguments": [], - "Options": [ - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Subcommands": [ - { - "Name": "list [options]", - "Description": "List jobs. Returns job key, state, process name, timing, and machine info. Use --state to filter by job state (Pending, Running, Successful, Faulted, Stopped, Stopping, Suspended, Resumed, Terminating). Provide --folder-path or --folder-key to scope to a folder, or omit for cross-folder listing." - }, - { - "Name": "get [options] ", - "Description": "Get a job by its key (GUID). Returns state, input/output arguments, timing, machine info, and error details if faulted. If the job is in Pending state, also includes troubleshooting reasons (robot availability, capacity, etc.). No folder context needed — the key is globally unique. Use 'jobs list' to find job keys." - }, - { - "Name": "start [options] ", - "Description": "Start a job for a process. Returns the created job(s) with keys. The is the GUID from 'processes list' (not 'packages list'). Folder is optional — if not provided, it is inferred from the process. Use 'packages entry-points' to see input/output argument schemas for the process." - }, - { - "Name": "stop [options] ", - "Description": "Stop one or more running or pending jobs. With a single key: respects --strategy (SoftStop or Kill). With multiple keys: always force-stops (Kill) using the bulk API. No folder context needed — resolved from job key." - }, - { - "Name": "restart [options] ", - "Description": "Restart a faulted or stopped job. Creates a new job from the same process. No folder context needed — resolved from job key." - }, - { - "Name": "resume [options] ", - "Description": "Resume a suspended job. Jobs become suspended when they use the 'Suspend' activity (long-running workflows that wait for external input). Optionally pass new --input-arguments to continue execution with updated data. No folder context needed — resolved from job key." - }, - { - "Name": "logs [options] ", - "Description": "Get robot execution logs for a job. Filter by --level to quickly find errors. Folder context is inferred from the job key. Logs are returned newest-first." - }, - { - "Name": "traces [options] ", - "Description": "Get LlmOps execution traces for an agentic job. Shows LLM interactions, tool calls, and agent decisions made during execution. Only available for Agent-type processes. Folder context is inferred from the job key." - }, - { - "Name": "healing-data [options] ", - "Description": "Download healing agent (Autopilot for Robots) recovery data as a ZIP file. Contains screenshots, UI element data, and recovery logs. Only available for jobs where the healing agent was active. Triggers an async export, polls until complete, then downloads the ZIP." - }, - { - "Name": "history [options] ", - "Description": "Get state transition history for a job. Shows timestamps for each state change (Pending → Running → Successful/Faulted). Useful for understanding job lifecycle and identifying delays. Folder context is resolved automatically from the job key." - }, - { - "Name": "help [command]", - "Description": "display help for command" - } - ], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -}{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "processes", - "Description": "Manage UiPath processes (runnable bindings of packages to folders). Processes are what you execute with 'jobs start'. Use 'packages' to manage the uploaded automation code.", - "Usage": "uip or processes [options] [command]", - "Arguments": [], - "Options": [ - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Subcommands": [ - { - "Name": "list [options]", - "Description": "List processes in a folder. Requires --folder-path or --folder-key. Returns process key (GUID), name, version, and type. Use the process key with 'jobs start' to execute a process." - }, - { - "Name": "get [options] ", - "Description": "Get detailed process information by key (GUID). Returns version, entry point, input/output argument schemas, process type, and configuration. No folder context needed — the key is globally unique. Use 'processes list' to find process keys." - }, - { - "Name": "create [options]", - "Description": "Create a new process by binding a package to a folder. Requires --folder-path or --folder-key. Use 'packages list' to find the package key and 'packages versions' to find available versions. After creation, start the process with 'jobs start '." - }, - { - "Name": "update-version [options] ", - "Description": "Update one or more processes to the latest package version, or a specific version with --version. Folder is optional — if not provided, it is inferred from the process key. Running jobs are not affected — only new jobs will use the updated version. Accepts one or more process keys. Multiple keys use the bulk API. Use --version only with a single key." - }, - { - "Name": "edit [options] ", - "Description": "Edit process settings by key (GUID). No folder needed — resolves cross-folder. Only provided fields are updated (PATCH). Use 'processes get' to see current values." - }, - { - "Name": "rollback [options] ", - "Description": "Rollback a process to its previous package version. Folder is optional — if not provided, it is inferred from the process key. Useful when a new version causes issues and you need to quickly revert." - }, - { - "Name": "help [command]", - "Description": "display help for command" - } - ], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/4bc4212220.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/4bc4212220.json deleted file mode 100644 index 309ccd31b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/4bc4212220.json +++ /dev/null @@ -1,100 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "or", - "Description": "Manage Orchestrator jobs, processes, packages, folders, machines, licenses, users, and permissions. Supports the full automation lifecycle: deploy packages, create processes, execute jobs, provision machines, manage licenses, manage access, and monitor results.", - "Usage": "uip or [options] [command]", - "Arguments": [], - "Options": [ - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Subcommands": [ - { - "Name": "attachments", - "Description": "Manage job attachments. Download files attached to jobs (screenshots, reports, data files). Attachments are tenant-scoped — no folder context needed." - }, - { - "Name": "audit-logs", - "Description": "View audit logs for the tenant. Shows who did what and when — tracks changes to users, processes, assets, jobs, and other entities." - }, - { - "Name": "calendars", - "Description": "Manage calendars (Non-Working Days). Calendars define excluded dates (holidays) used by time triggers to skip scheduled runs." - }, - { - "Name": "credential-stores", - "Description": "Manage Orchestrator credential stores (tenant-scoped secure storage backends for credentials and assets)." - }, - { - "Name": "feeds", - "Description": "Manage UiPath package feeds. Feeds are repositories that store automation packages (.nupkg). Use the feed ID with 'packages list --feed-id' to browse packages in a specific feed." - }, - { - "Name": "folders", - "Description": "Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder." - }, - { - "Name": "jobs", - "Description": "Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces." - }, - { - "Name": "licenses", - "Description": "Manage UiPath Orchestrator licenses. View license allocation and usage across robot types, list runtime and named-user license assignments, and toggle machine licensing on or off." - }, - { - "Name": "machines", - "Description": "Manage UiPath Orchestrator machines (robot host definitions). Machines define the hosts where robots execute automations, along with their license slot allocations. Machines are tenant-scoped but can be assigned to specific folders." - }, - { - "Name": "packages", - "Description": "Manage UiPath automation packages (.nupkg) in the Orchestrator feed. Packages are uploaded code artifacts. Use 'processes' to manage runnable process bindings in folders." - }, - { - "Name": "roles", - "Description": "Manage Orchestrator roles and permissions (RBAC). Roles group permissions and are assigned to users at the tenant or folder level. Tenant roles apply globally; Folder roles scope access to specific folders. Use 'roles list-roles' to find role keys, 'roles list-permissions' for grantable permissions." - }, - { - "Name": "sessions", - "Description": "View and manage Orchestrator sessions (active host connections). Sessions represent live connections between hosts and Orchestrator, including their current state, runtime type, and maintenance status. Use subcommands to list sessions, inspect machine runtimes, toggle debug mode, delete inactive sessions, or set maintenance mode." - }, - { - "Name": "processes", - "Description": "Manage UiPath processes (runnable bindings of packages to folders). Processes are what you execute with 'jobs start'. Use 'packages' to manage the uploaded automation code." - }, - { - "Name": "settings", - "Description": "Manage tenant settings. Settings are key-value pairs — use 'settings list' to discover keys. Categories and common keys: [General] Abp.Timing.TimeZone (timezone ID, e.g. 'UTC' — use 'settings timezones' to list valid IDs), Abp.Localization.DefaultLanguageName (language code, e.g. 'en', 'fr', 'de'). [Email] Alerts.Email.Enabled (true/false), Abp.Net.Mail.Smtp.Host (hostname), .Port (integer, e.g. 587), .UserName, .Password, .EnableSsl (true/false), .UseDefaultCredentials (true/false), Abp.Net.Mail.DefaultFromAddress (email address). [Robot Security] Auth.Robot.EnforceAuthenticationToken (true/false — force token auth for robots), Auth.UnattendedRobot.ForceClientCredentials (True/False), AttendedRobot.RunDisconnectedHours (integer, 0=disabled, hours a robot can run disconnected). [Deployment] DeploymentUrl (NuGet feed URL), NuGet.Packages.ApiKey (API key string), Deployment.Libraries.Url, NuGet.Activities.ApiKey, Deployment.Processes.AuthenticationType (Secure/apiKey). [Scalability] Scalability.SignalR.Enabled (true/false), Scalability.SignalR.Transport (bitmask: 1=WebSockets, 2=SSE, 4=LongPolling, 7=all), Scalability.Heartbeat.PeriodSeconds (integer, default 30, seconds between robot heartbeats), Scalability.Heartbeat.FailureThreshold (integer, default 4, missed heartbeats before marking not responding). [Triggers] Triggers.DisableWhenFailedCount (integer, default 10, consecutive failures to auto-disable), Triggers.DisableWhenFailingSinceDays (integer, default 1), Triggers.JobsCountStrategy (PERPROCESS/PERTRIGGER), QueueTrigger.ActivationFrequencyInMinutes (integer, default 30), ConnectedTriggers.AutoDisableJobFailuresThreshold (integer, default 5). [Queue] inProgressMaxNumberOfMinutes (integer, default 1440=24h, max time an item stays InProgress). [Jobs] Jobs.TerminatingJobsTimeout (integer, default 1440 minutes=24h, timeout for terminating jobs). Use 'settings execution' for structured execution settings with display names and allowed values." - }, - { - "Name": "users", - "Description": "Manage Orchestrator users (tenant-scoped). List, create, edit, and delete users. Assign users to folders with specific roles. Manage tenant-level role assignments. Use 'users list' to find user keys (GUIDs)." - }, - { - "Name": "help [command]", - "Description": "display help for command" - } - ], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/667509c877.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/667509c877.json deleted file mode 100644 index 240ffbd4e1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/667509c877.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Successful", - "Timestamp": "2026-05-04T10:39:28.857Z", - "JobId": 670881758 - }, - { - "State": "Running", - "Timestamp": "2026-05-04T10:31:31.200Z", - "JobId": 670881758 - }, - { - "State": "Pending", - "Timestamp": "2026-05-04T10:31:30.837Z", - "JobId": 670881758 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/667cf1dcb5.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/667cf1dcb5.json deleted file mode 100644 index 91dceacf38..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/667cf1dcb5.json +++ /dev/null @@ -1,155 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "start", - "Description": "Start a job for a process. Returns the created job(s) with keys. The is the GUID from 'processes list' (not 'packages list'). Folder is optional — if not provided, it is inferred from the process. Use 'packages entry-points' to see input/output argument schemas for the process.", - "Usage": "uip or jobs start [options] ", - "Arguments": [ - { - "Name": "process-key", - "Description": "Process key (GUID) from 'processes list'. Identifies which process to run.", - "Required": true - } - ], - "Options": [ - { - "Flags": "-t, --tenant ", - "Description": "Tenant (optional, defaults to value selected during auth)" - }, - { - "Flags": "--folder-path ", - "Description": "Folder path (optional — inferred from process if not provided)" - }, - { - "Flags": "--folder-key ", - "Description": "Folder key (optional — inferred from process if not provided)" - }, - { - "Flags": "--strategy ", - "Description": "Execution strategy (ModernJobsCount, All, Specific, JobsCount)", - "Default": "ModernJobsCount" - }, - { - "Flags": "--jobs-count ", - "Description": "Number of jobs to create (for JobsCount/ModernJobsCount strategy)", - "Default": "1" - }, - { - "Flags": "--input-arguments ", - "Description": "Input arguments as JSON string (mutually exclusive with --input-file). Auto-offloads to InputFile if >10K chars. Use 'packages entry-points' to see the expected schema." - }, - { - "Flags": "--input-file ", - "Description": "Upload a file as the job's InputFile (mutually exclusive with --input-arguments). The file is uploaded as an attachment and its ID is set on the InputFile field." - }, - { - "Flags": "--attachment <[name=]path>", - "Description": "Upload a file and attach it to the job (repeatable). Format: path (auto-names from filename) or name=path (explicit name). Example: --attachment ./doc.pdf --attachment invoice=./report.pdf" - }, - { - "Flags": "--attachment-id ", - "Description": "Attach an existing attachment by ID (repeatable). Use 'attachments list --job-key ' to find IDs from a previous job." - }, - { - "Flags": "--runtime-type ", - "Description": "Execution runtime (Unattended, Headless, Serverless, NonProduction, Development, TestAutomation). Defaults to folder configuration." - }, - { - "Flags": "--user-keys ", - "Description": "Comma-separated user keys (GUIDs) to run as. Use 'users list' to find keys." - }, - { - "Flags": "--machine-keys ", - "Description": "Comma-separated machine keys (GUIDs) to target. Use 'machines list' to find keys." - }, - { - "Flags": "--environment-variables ", - "Description": "Environment variables as JSON string" - }, - { - "Flags": "--run-as-me", - "Description": "Run job under the caller's identity" - }, - { - "Flags": "--job-priority ", - "Description": "Job priority (Low, Normal, High)" - }, - { - "Flags": "--reference ", - "Description": "User-specified reference for the job" - }, - { - "Flags": "--wait-for-completion", - "Description": "Poll until the job reaches a terminal state (Successful, Faulted, Stopped) and return its output" - }, - { - "Flags": "--timeout ", - "Description": "Maximum time to wait for completion in seconds (default: 300)", - "Default": "300" - }, - { - "Flags": "--poll-interval ", - "Description": "Seconds between status checks (default: 5)", - "Default": "5" - }, - { - "Flags": "--output-dir ", - "Description": "Directory for downloading OutputFile (default: current directory). Only with --wait-for-completion." - }, - { - "Flags": "--no-download", - "Description": "Skip automatic OutputFile download. Only with --wait-for-completion." - }, - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Examples": [ - { - "Description": "Start a process by its key", - "Command": "uip or jobs start c3d4e5f6-0000-0000-0000-000000000001 --jobs-count 1", - "Output": { - "Code": "JobStarted", - "Data": { - "Jobs": [ - { - "Key": "b2c3d4e5-0000-0000-0000-000000000099", - "State": "Pending", - "ProcessName": "InvoiceProcessing", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2025-04-15T10:29:55Z", - "Id": 1099 - } - ] - } - } - } - ], - "Subcommands": [], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/924fb2bc90.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/924fb2bc90.json deleted file mode 100644 index 8546a32fc5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/924fb2bc90.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "0cf4a4c8-9ada-4189-bf2d-a01d8edc18c7", - "TraceId": "777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "ParentId": null, - "Name": "Robot Job 777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "StartTime": "2026-05-04T10:31:32.2911004Z", - "EndTime": "2026-05-04T10:39:29.0035839Z", - "Attributes": "{\"Exception\":null}", - "Status": 1, - "OrganizationId": "d308a0c0-d09e-4158-a4e8-17d099552d6d", - "TenantId": "71fffd7f-0605-43ba-b025-20226cf149ce", - "ExpiryTimeUtc": null, - "FolderKey": "445c18c8-3f11-43b6-ad44-6ff83f33af18", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "63ad34e1-95ec-4b2f-aeea-34883e185220", - "JobKey": "777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-05-04T10:39:28.6501777Z", - "AgentVersion": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/README.md deleted file mode 100644 index 60ae3a39ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Rpa Selector Healing Disabled — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause: Benefits Enrollment Portal UI Change — Employee Name Input Field Selector Broken** - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name rpa-selector-healing-disabled --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/c98d469724.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/c98d469724.json deleted file mode 100644 index 151ab0dc2c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/c98d469724.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "StartTime": "2026-05-04T10:31:31.190Z", - "EndTime": "2026-05-04T10:39:28.850Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "01610aff-fe36-4a9f-a26d-f67287fe7915", - "Info": "Job completed", - "CreationTime": "2026-05-04T10:31:30.833Z", - "ReleaseName": "BenefitsEnrollment", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 4669692, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 7792378, - "FolderKey": "445c18c8-3f11-43b6-ad44-6ff83f33af18", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-04T10:39:28.850Z", - "TraceId": "777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 670881758 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/e5823c6244.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/e5823c6244.json deleted file mode 100644 index 9797c946a5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/e5823c6244.json +++ /dev/null @@ -1,156 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:39:25.379Z", - "Message": "Failed to enroll Kevin Park : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:39:09.653Z", - "Message": "Failed to enroll Laura Sanchez : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:38:53.978Z", - "Message": "Failed to enroll Tyler Nguyen : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:38:38.389Z", - "Message": "Failed to enroll Aisha Nkomo : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:38:22.685Z", - "Message": "Failed to enroll James O'Brien : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:38:07.097Z", - "Message": "Failed to enroll Priya Patel : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:37:51.491Z", - "Message": "Failed to enroll Marcus Williams : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:37:35.716Z", - "Message": "Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:37:20.050Z", - "Message": "Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:37:04.421Z", - "Message": "Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:36:48.822Z", - "Message": "Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:36:32.962Z", - "Message": "Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:36:17.122Z", - "Message": "Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:36:01.323Z", - "Message": "Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:35:45.546Z", - "Message": "Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:35:29.853Z", - "Message": "Failed to enroll Kevin Park : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:35:14.248Z", - "Message": "Failed to enroll Laura Sanchez : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:34:58.561Z", - "Message": "Failed to enroll Tyler Nguyen : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:34:42.847Z", - "Message": "Failed to enroll Aisha Nkomo : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:34:27.166Z", - "Message": "Failed to enroll James O'Brien : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:34:11.557Z", - "Message": "Failed to enroll Priya Patel : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:33:55.763Z", - "Message": "Failed to enroll Marcus Williams : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:33:40.025Z", - "Message": "Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:33:24.331Z", - "Message": "Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:33:08.675Z", - "Message": "Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:32:52.948Z", - "Message": "Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:32:37.279Z", - "Message": "Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:32:21.686Z", - "Message": "Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:32:06.058Z", - "Message": "Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T10:31:50.109Z", - "Message": "Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\r\n[1] \n[2] \n\r\nSearch failed at selector tag:\r\n[2] \r\n\r\nThe closest matches found are:\r\n[50%] \n[50%] \n[50%] " - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/manifest.json deleted file mode 100644 index 8815682375..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/m/r/manifest.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py — shell-glue + --output json suffixes removed from match strings (broader substring match); fixtures backfilled from .local/investigations/raw/.", - "rules": [ - { - "match": "or jobs start --help", - "file": "667cf1dcb5.json" - }, - { - "match": "or processes update --help", - "file": "269b3bd27b.json" - }, - { - "match": "or jobs --help", - "file": "3ed41069b3.json" - }, - { - "match": "or --help", - "file": "4bc4212220.json" - }, - { - "match": "or jobs get 777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "file": "c98d469724.json" - }, - { - "match": "or jobs logs 777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "file": "e5823c6244.json" - }, - { - "match": "or jobs history 777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "file": "667509c877.json" - }, - { - "match": "or jobs traces 777d35d4-26bc-49ad-bbdb-5243ba2947fc", - "file": "924fb2bc90.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/uip-fixture.json new file mode 100644 index 0000000000..0a2f9ad9e8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/data/uip-fixture.json @@ -0,0 +1,94 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "start", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"start\",\r\n \"Description\": \"Start a job for a process. Returns the created job(s) with keys. The is the GUID from 'processes list' (not 'packages list'). Folder is optional — if not provided, it is inferred from the process. Use 'packages entry-points' to see input/output argument schemas for the process.\",\r\n \"Usage\": \"uip or jobs start [options] \",\r\n \"Arguments\": [\r\n {\r\n \"Name\": \"process-key\",\r\n \"Description\": \"Process key (GUID) from 'processes list'. Identifies which process to run.\",\r\n \"Required\": true\r\n }\r\n ],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-t, --tenant \",\r\n \"Description\": \"Tenant (optional, defaults to value selected during auth)\"\r\n },\r\n {\r\n \"Flags\": \"--folder-path \",\r\n \"Description\": \"Folder path (optional — inferred from process if not provided)\"\r\n },\r\n {\r\n \"Flags\": \"--folder-key \",\r\n \"Description\": \"Folder key (optional — inferred from process if not provided)\"\r\n },\r\n {\r\n \"Flags\": \"--strategy \",\r\n \"Description\": \"Execution strategy (ModernJobsCount, All, Specific, JobsCount)\",\r\n \"Default\": \"ModernJobsCount\"\r\n },\r\n {\r\n \"Flags\": \"--jobs-count \",\r\n \"Description\": \"Number of jobs to create (for JobsCount/ModernJobsCount strategy)\",\r\n \"Default\": \"1\"\r\n },\r\n {\r\n \"Flags\": \"--input-arguments \",\r\n \"Description\": \"Input arguments as JSON string (mutually exclusive with --input-file). Auto-offloads to InputFile if >10K chars. Use 'packages entry-points' to see the expected schema.\"\r\n },\r\n {\r\n \"Flags\": \"--input-file \",\r\n \"Description\": \"Upload a file as the job's InputFile (mutually exclusive with --input-arguments). The file is uploaded as an attachment and its ID is set on the InputFile field.\"\r\n },\r\n {\r\n \"Flags\": \"--attachment <[name=]path>\",\r\n \"Description\": \"Upload a file and attach it to the job (repeatable). Format: path (auto-names from filename) or name=path (explicit name). Example: --attachment ./doc.pdf --attachment invoice=./report.pdf\"\r\n },\r\n {\r\n \"Flags\": \"--attachment-id \",\r\n \"Description\": \"Attach an existing attachment by ID (repeatable). Use 'attachments list --job-key ' to find IDs from a previous job.\"\r\n },\r\n {\r\n \"Flags\": \"--runtime-type \",\r\n \"Description\": \"Execution runtime (Unattended, Headless, Serverless, NonProduction, Development, TestAutomation). Defaults to folder configuration.\"\r\n },\r\n {\r\n \"Flags\": \"--user-keys \",\r\n \"Description\": \"Comma-separated user keys (GUIDs) to run as. Use 'users list' to find keys.\"\r\n },\r\n {\r\n \"Flags\": \"--machine-keys \",\r\n \"Description\": \"Comma-separated machine keys (GUIDs) to target. Use 'machines list' to find keys.\"\r\n },\r\n {\r\n \"Flags\": \"--environment-variables \",\r\n \"Description\": \"Environment variables as JSON string\"\r\n },\r\n {\r\n \"Flags\": \"--run-as-me\",\r\n \"Description\": \"Run job under the caller's identity\"\r\n },\r\n {\r\n \"Flags\": \"--job-priority \",\r\n \"Description\": \"Job priority (Low, Normal, High)\"\r\n },\r\n {\r\n \"Flags\": \"--reference \",\r\n \"Description\": \"User-specified reference for the job\"\r\n },\r\n {\r\n \"Flags\": \"--wait-for-completion\",\r\n \"Description\": \"Poll until the job reaches a terminal state (Successful, Faulted, Stopped) and return its output\"\r\n },\r\n {\r\n \"Flags\": \"--timeout \",\r\n \"Description\": \"Maximum time to wait for completion in seconds (default: 300)\",\r\n \"Default\": \"300\"\r\n },\r\n {\r\n \"Flags\": \"--poll-interval \",\r\n \"Description\": \"Seconds between status checks (default: 5)\",\r\n \"Default\": \"5\"\r\n },\r\n {\r\n \"Flags\": \"--output-dir \",\r\n \"Description\": \"Directory for downloading OutputFile (default: current directory). Only with --wait-for-completion.\"\r\n },\r\n {\r\n \"Flags\": \"--no-download\",\r\n \"Description\": \"Skip automatic OutputFile download. Only with --wait-for-completion.\"\r\n },\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Examples\": [\r\n {\r\n \"Description\": \"Start a process by its key\",\r\n \"Command\": \"uip or jobs start c3d4e5f6-0000-0000-0000-000000000001 --jobs-count 1\",\r\n \"Output\": {\r\n \"Code\": \"JobStarted\",\r\n \"Data\": {\r\n \"Jobs\": [\r\n {\r\n \"Key\": \"b2c3d4e5-0000-0000-0000-000000000099\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"InvoiceProcessing\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2025-04-15T10:29:55Z\",\r\n \"Id\": 1099\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n ],\r\n \"Subcommands\": [],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}" + }, + { + "argv": [ + "or", + "processes", + "update", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"update\",\r\n \"Description\": \"Update process settings by key (GUID). No folder needed — resolves cross-folder. Only provided fields are updated (PATCH). Use 'processes get' to see current values.\",\r\n \"Usage\": \"uip or processes update [options] \",\r\n \"Arguments\": [\r\n {\r\n \"Name\": \"process-key\",\r\n \"Description\": \"Process key (GUID)\",\r\n \"Required\": true\r\n }\r\n ],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-t, --tenant \",\r\n \"Description\": \"Tenant (optional, defaults to value selected during auth)\"\r\n },\r\n {\r\n \"Flags\": \"-n, --name \",\r\n \"Description\": \"Process display name\"\r\n },\r\n {\r\n \"Flags\": \"-d, --description \",\r\n \"Description\": \"Process description\"\r\n },\r\n {\r\n \"Flags\": \"--entry-point \",\r\n \"Description\": \"Entry point workflow path (for multi-entry-point packages)\"\r\n },\r\n {\r\n \"Flags\": \"--input-arguments \",\r\n \"Description\": \"Default input arguments as JSON string\"\r\n },\r\n {\r\n \"Flags\": \"--job-priority \",\r\n \"Description\": \"Default job priority (Low, Normal, High)\"\r\n },\r\n {\r\n \"Flags\": \"--specific-priority \",\r\n \"Description\": \"Specific priority value (1-100). Mutually exclusive with --job-priority.\"\r\n },\r\n {\r\n \"Flags\": \"--auto-update\",\r\n \"Description\": \"Enable auto-update to latest package version\"\r\n },\r\n {\r\n \"Flags\": \"--no-auto-update\",\r\n \"Description\": \"Disable auto-update to latest package version\"\r\n },\r\n {\r\n \"Flags\": \"--hidden-for-attended\",\r\n \"Description\": \"Hide from attended robot users\"\r\n },\r\n {\r\n \"Flags\": \"--visible-for-attended\",\r\n \"Description\": \"Show to attended robot users\"\r\n },\r\n {\r\n \"Flags\": \"--auto-create-triggers\",\r\n \"Description\": \"Auto-create connected triggers on deploy\"\r\n },\r\n {\r\n \"Flags\": \"--no-auto-create-triggers\",\r\n \"Description\": \"Disable auto-create connected triggers on deploy\"\r\n },\r\n {\r\n \"Flags\": \"--retention-period \",\r\n \"Description\": \"Job retention period in days (1–180)\"\r\n },\r\n {\r\n \"Flags\": \"--retention-action \",\r\n \"Description\": \"Retention action (Delete, Archive, None)\"\r\n },\r\n {\r\n \"Flags\": \"--retention-bucket \",\r\n \"Description\": \"Storage bucket ID for archived jobs (required when action is Archive)\"\r\n },\r\n {\r\n \"Flags\": \"--stale-retention-period \",\r\n \"Description\": \"Stale job retention period in days\"\r\n },\r\n {\r\n \"Flags\": \"--stale-retention-action \",\r\n \"Description\": \"Stale retention action (Delete, Archive, None)\"\r\n },\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Examples\": [\r\n {\r\n \"Description\": \"Change a process description\",\r\n \"Command\": \"uip or processes update c3d4e5f6-0000-0000-0000-000000000001 --description \\\"Monthly invoice batch\\\"\",\r\n \"Output\": {\r\n \"Code\": \"ProcessUpdated\",\r\n \"Data\": {\r\n \"Key\": \"c3d4e5f6-0000-0000-0000-000000000001\",\r\n \"Name\": \"InvoiceProcessing\",\r\n \"Status\": \"Updated successfully\"\r\n }\r\n }\r\n }\r\n ],\r\n \"Subcommands\": [],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "get", + "777d35d4-26bc-49ad-bbdb-5243ba2947fc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"777d35d4-26bc-49ad-bbdb-5243ba2947fc\",\r\n \"StartTime\": \"2026-05-04T10:31:31.190Z\",\r\n \"EndTime\": \"2026-05-04T10:39:28.850Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"01610aff-fe36-4a9f-a26d-f67287fe7915\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-05-04T10:31:30.833Z\",\r\n \"ReleaseName\": \"BenefitsEnrollment\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 4669692,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 7792378,\r\n \"FolderKey\": \"445c18c8-3f11-43b6-ad44-6ff83f33af18\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-04T10:39:28.850Z\",\r\n \"TraceId\": \"777d35d4-26bc-49ad-bbdb-5243ba2947fc\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 670881758\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "777d35d4-26bc-49ad-bbdb-5243ba2947fc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:39:25.379Z\",\r\n \"Message\": \"Failed to enroll Kevin Park : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:39:09.653Z\",\r\n \"Message\": \"Failed to enroll Laura Sanchez : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:38:53.978Z\",\r\n \"Message\": \"Failed to enroll Tyler Nguyen : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:38:38.389Z\",\r\n \"Message\": \"Failed to enroll Aisha Nkomo : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:38:22.685Z\",\r\n \"Message\": \"Failed to enroll James O'Brien : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:38:07.097Z\",\r\n \"Message\": \"Failed to enroll Priya Patel : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:37:51.491Z\",\r\n \"Message\": \"Failed to enroll Marcus Williams : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:37:35.716Z\",\r\n \"Message\": \"Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:37:20.050Z\",\r\n \"Message\": \"Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:37:04.421Z\",\r\n \"Message\": \"Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:36:48.822Z\",\r\n \"Message\": \"Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:36:32.962Z\",\r\n \"Message\": \"Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:36:17.122Z\",\r\n \"Message\": \"Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:36:01.323Z\",\r\n \"Message\": \"Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:35:45.546Z\",\r\n \"Message\": \"Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:35:29.853Z\",\r\n \"Message\": \"Failed to enroll Kevin Park : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:35:14.248Z\",\r\n \"Message\": \"Failed to enroll Laura Sanchez : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:34:58.561Z\",\r\n \"Message\": \"Failed to enroll Tyler Nguyen : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:34:42.847Z\",\r\n \"Message\": \"Failed to enroll Aisha Nkomo : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:34:27.166Z\",\r\n \"Message\": \"Failed to enroll James O'Brien : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:34:11.557Z\",\r\n \"Message\": \"Failed to enroll Priya Patel : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:33:55.763Z\",\r\n \"Message\": \"Failed to enroll Marcus Williams : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:33:40.025Z\",\r\n \"Message\": \"Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:33:24.331Z\",\r\n \"Message\": \"Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:33:08.675Z\",\r\n \"Message\": \"Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:32:52.948Z\",\r\n \"Message\": \"Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:32:37.279Z\",\r\n \"Message\": \"Failed to enroll Emma Thompson : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:32:21.686Z\",\r\n \"Message\": \"Failed to enroll David Osei : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:32:06.058Z\",\r\n \"Message\": \"Failed to enroll Sarah Kim : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T10:31:50.109Z\",\r\n \"Message\": \"Failed to enroll John Rivera : Could not find the UI element corresponding to this selector:\\r\\n[1] \\n[2] \\n\\r\\nSearch failed at selector tag:\\r\\n[2] \\r\\n\\r\\nThe closest matches found are:\\r\\n[50%] \\n[50%] \\n[50%] \"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "777d35d4-26bc-49ad-bbdb-5243ba2947fc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Successful\",\r\n \"Timestamp\": \"2026-05-04T10:39:28.857Z\",\r\n \"JobId\": 670881758\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-04T10:31:31.200Z\",\r\n \"JobId\": 670881758\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-04T10:31:30.837Z\",\r\n \"JobId\": 670881758\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "777d35d4-26bc-49ad-bbdb-5243ba2947fc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"0cf4a4c8-9ada-4189-bf2d-a01d8edc18c7\",\r\n \"TraceId\": \"777d35d4-26bc-49ad-bbdb-5243ba2947fc\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 777d35d4-26bc-49ad-bbdb-5243ba2947fc\",\r\n \"StartTime\": \"2026-05-04T10:31:32.2911004Z\",\r\n \"EndTime\": \"2026-05-04T10:39:29.0035839Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":null}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"d308a0c0-d09e-4158-a4e8-17d099552d6d\",\r\n \"TenantId\": \"71fffd7f-0605-43ba-b025-20226cf149ce\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"445c18c8-3f11-43b6-ad44-6ff83f33af18\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"63ad34e1-95ec-4b2f-aeea-34883e185220\",\r\n \"JobKey\": \"777d35d4-26bc-49ad-bbdb-5243ba2947fc\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-05-04T10:39:28.6501777Z\",\r\n \"AgentVersion\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"jobs\",\r\n \"Description\": \"Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces.\",\r\n \"Usage\": \"uip or jobs [options] [command]\",\r\n \"Arguments\": [],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Subcommands\": [\r\n {\r\n \"Name\": \"list [options]\",\r\n \"Description\": \"List jobs. Returns job key, state, process name, timing, and machine info. Use --state to filter by job state (Pending, Running, Successful, Faulted, Stopped, Stopping, Suspended, Resumed, Terminating). Provide --folder-path or --folder-key to scope to a folder, or omit for cross-folder listing.\"\r\n },\r\n {\r\n \"Name\": \"get [options] \",\r\n \"Description\": \"Get a job by its key (GUID). Returns state, input/output arguments, timing, machine info, and error details if faulted. If the job is in Pending state, also includes troubleshooting reasons (robot availability, capacity, etc.). No folder context needed — the key is globally unique. Use 'jobs list' to find job keys.\"\r\n },\r\n {\r\n \"Name\": \"start [options] \",\r\n \"Description\": \"Start a job for a process. Returns the created job(s) with keys. The is the GUID from 'processes list' (not 'packages list'). Folder is optional — if not provided, it is inferred from the process. Use 'packages entry-points' to see input/output argument schemas for the process.\"\r\n },\r\n {\r\n \"Name\": \"stop [options] \",\r\n \"Description\": \"Stop one or more running or pending jobs. With a single key: respects --strategy (SoftStop or Kill). With multiple keys: always force-stops (Kill) using the bulk API. No folder context needed — resolved from job key.\"\r\n },\r\n {\r\n \"Name\": \"restart [options] \",\r\n \"Description\": \"Restart a faulted or stopped job. Creates a new job from the same process. No folder context needed — resolved from job key.\"\r\n },\r\n {\r\n \"Name\": \"resume [options] \",\r\n \"Description\": \"Resume a suspended job. Jobs become suspended when they use the 'Suspend' activity (long-running workflows that wait for external input). Optionally pass new --input-arguments to continue execution with updated data. No folder context needed — resolved from job key.\"\r\n },\r\n {\r\n \"Name\": \"logs [options] \",\r\n \"Description\": \"Get robot execution logs for a job. Filter by --level to quickly find errors. Folder context is inferred from the job key. Logs are returned newest-first.\"\r\n },\r\n {\r\n \"Name\": \"traces [options] \",\r\n \"Description\": \"Get LlmOps execution traces for an agentic job. Shows LLM interactions, tool calls, and agent decisions made during execution. Only available for Agent-type processes. Folder context is inferred from the job key.\"\r\n },\r\n {\r\n \"Name\": \"healing-data [options] \",\r\n \"Description\": \"Download healing agent (Autopilot for Robots) recovery data as a ZIP file. Contains screenshots, UI element data, and recovery logs. Only available for jobs where the healing agent was active. Triggers an async export, polls until complete, then downloads the ZIP.\"\r\n },\r\n {\r\n \"Name\": \"history [options] \",\r\n \"Description\": \"Get state transition history for a job. Shows timestamps for each state change (Pending → Running → Successful/Faulted). Useful for understanding job lifecycle and identifying delays. Folder context is resolved automatically from the job key.\"\r\n },\r\n {\r\n \"Name\": \"help [command]\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"processes\",\r\n \"Description\": \"Manage UiPath processes (runnable bindings of packages to folders). Processes are what you execute with 'jobs start'. Use 'packages' to manage the uploaded automation code.\",\r\n \"Usage\": \"uip or processes [options] [command]\",\r\n \"Arguments\": [],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Subcommands\": [\r\n {\r\n \"Name\": \"list [options]\",\r\n \"Description\": \"List processes in a folder. Requires --folder-path or --folder-key. Returns process key (GUID), name, version, and type. Use the process key with 'jobs start' to execute a process.\"\r\n },\r\n {\r\n \"Name\": \"get [options] \",\r\n \"Description\": \"Get detailed process information by key (GUID). Returns version, entry point, input/output argument schemas, process type, and configuration. No folder context needed — the key is globally unique. Use 'processes list' to find process keys.\"\r\n },\r\n {\r\n \"Name\": \"create [options]\",\r\n \"Description\": \"Create a new process by binding a package to a folder. Requires --folder-path or --folder-key. Use 'packages list' to find the package key and 'packages versions' to find available versions. After creation, start the process with 'jobs start '.\"\r\n },\r\n {\r\n \"Name\": \"update-version [options] \",\r\n \"Description\": \"Update one or more processes to the latest package version, or a specific version with --version. Folder is optional — if not provided, it is inferred from the process key. Running jobs are not affected — only new jobs will use the updated version. Accepts one or more process keys. Multiple keys use the bulk API. Use --version only with a single key.\"\r\n },\r\n {\r\n \"Name\": \"edit [options] \",\r\n \"Description\": \"Edit process settings by key (GUID). No folder needed — resolves cross-folder. Only provided fields are updated (PATCH). Use 'processes get' to see current values.\"\r\n },\r\n {\r\n \"Name\": \"rollback [options] \",\r\n \"Description\": \"Rollback a process to its previous package version. Folder is optional — if not provided, it is inferred from the process key. Useful when a new version causes issues and you need to quickly revert.\"\r\n },\r\n {\r\n \"Name\": \"help [command]\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}" + }, + { + "argv": [ + "or", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"or\",\r\n \"Description\": \"Manage Orchestrator jobs, processes, packages, folders, machines, licenses, users, and permissions. Supports the full automation lifecycle: deploy packages, create processes, execute jobs, provision machines, manage licenses, manage access, and monitor results.\",\r\n \"Usage\": \"uip or [options] [command]\",\r\n \"Arguments\": [],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Subcommands\": [\r\n {\r\n \"Name\": \"attachments\",\r\n \"Description\": \"Manage job attachments. Download files attached to jobs (screenshots, reports, data files). Attachments are tenant-scoped — no folder context needed.\"\r\n },\r\n {\r\n \"Name\": \"audit-logs\",\r\n \"Description\": \"View audit logs for the tenant. Shows who did what and when — tracks changes to users, processes, assets, jobs, and other entities.\"\r\n },\r\n {\r\n \"Name\": \"calendars\",\r\n \"Description\": \"Manage calendars (Non-Working Days). Calendars define excluded dates (holidays) used by time triggers to skip scheduled runs.\"\r\n },\r\n {\r\n \"Name\": \"credential-stores\",\r\n \"Description\": \"Manage Orchestrator credential stores (tenant-scoped secure storage backends for credentials and assets).\"\r\n },\r\n {\r\n \"Name\": \"feeds\",\r\n \"Description\": \"Manage UiPath package feeds. Feeds are repositories that store automation packages (.nupkg). Use the feed ID with 'packages list --feed-id' to browse packages in a specific feed.\"\r\n },\r\n {\r\n \"Name\": \"folders\",\r\n \"Description\": \"Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder.\"\r\n },\r\n {\r\n \"Name\": \"jobs\",\r\n \"Description\": \"Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces.\"\r\n },\r\n {\r\n \"Name\": \"licenses\",\r\n \"Description\": \"Manage UiPath Orchestrator licenses. View license allocation and usage across robot types, list runtime and named-user license assignments, and toggle machine licensing on or off.\"\r\n },\r\n {\r\n \"Name\": \"machines\",\r\n \"Description\": \"Manage UiPath Orchestrator machines (robot host definitions). Machines define the hosts where robots execute automations, along with their license slot allocations. Machines are tenant-scoped but can be assigned to specific folders.\"\r\n },\r\n {\r\n \"Name\": \"packages\",\r\n \"Description\": \"Manage UiPath automation packages (.nupkg) in the Orchestrator feed. Packages are uploaded code artifacts. Use 'processes' to manage runnable process bindings in folders.\"\r\n },\r\n {\r\n \"Name\": \"roles\",\r\n \"Description\": \"Manage Orchestrator roles and permissions (RBAC). Roles group permissions and are assigned to users at the tenant or folder level. Tenant roles apply globally; Folder roles scope access to specific folders. Use 'roles list-roles' to find role keys, 'roles list-permissions' for grantable permissions.\"\r\n },\r\n {\r\n \"Name\": \"sessions\",\r\n \"Description\": \"View and manage Orchestrator sessions (active host connections). Sessions represent live connections between hosts and Orchestrator, including their current state, runtime type, and maintenance status. Use subcommands to list sessions, inspect machine runtimes, toggle debug mode, delete inactive sessions, or set maintenance mode.\"\r\n },\r\n {\r\n \"Name\": \"processes\",\r\n \"Description\": \"Manage UiPath processes (runnable bindings of packages to folders). Processes are what you execute with 'jobs start'. Use 'packages' to manage the uploaded automation code.\"\r\n },\r\n {\r\n \"Name\": \"settings\",\r\n \"Description\": \"Manage tenant settings. Settings are key-value pairs — use 'settings list' to discover keys. Categories and common keys: [General] Abp.Timing.TimeZone (timezone ID, e.g. 'UTC' — use 'settings timezones' to list valid IDs), Abp.Localization.DefaultLanguageName (language code, e.g. 'en', 'fr', 'de'). [Email] Alerts.Email.Enabled (true/false), Abp.Net.Mail.Smtp.Host (hostname), .Port (integer, e.g. 587), .UserName, .Password, .EnableSsl (true/false), .UseDefaultCredentials (true/false), Abp.Net.Mail.DefaultFromAddress (email address). [Robot Security] Auth.Robot.EnforceAuthenticationToken (true/false — force token auth for robots), Auth.UnattendedRobot.ForceClientCredentials (True/False), AttendedRobot.RunDisconnectedHours (integer, 0=disabled, hours a robot can run disconnected). [Deployment] DeploymentUrl (NuGet feed URL), NuGet.Packages.ApiKey (API key string), Deployment.Libraries.Url, NuGet.Activities.ApiKey, Deployment.Processes.AuthenticationType (Secure/apiKey). [Scalability] Scalability.SignalR.Enabled (true/false), Scalability.SignalR.Transport (bitmask: 1=WebSockets, 2=SSE, 4=LongPolling, 7=all), Scalability.Heartbeat.PeriodSeconds (integer, default 30, seconds between robot heartbeats), Scalability.Heartbeat.FailureThreshold (integer, default 4, missed heartbeats before marking not responding). [Triggers] Triggers.DisableWhenFailedCount (integer, default 10, consecutive failures to auto-disable), Triggers.DisableWhenFailingSinceDays (integer, default 1), Triggers.JobsCountStrategy (PERPROCESS/PERTRIGGER), QueueTrigger.ActivationFrequencyInMinutes (integer, default 30), ConnectedTriggers.AutoDisableJobFailuresThreshold (integer, default 5). [Queue] inProgressMaxNumberOfMinutes (integer, default 1440=24h, max time an item stays InProgress). [Jobs] Jobs.TerminatingJobsTimeout (integer, default 1440 minutes=24h, timeout for terminating jobs). Use 'settings execution' for structured execution settings with display names and allowed values.\"\r\n },\r\n {\r\n \"Name\": \"users\",\r\n \"Description\": \"Manage Orchestrator users (tenant-scoped). List, create, edit, and delete users. Assign users to folders with specific roles. Manage tenant-level role assignments. Use 'users list' to find user keys (GUIDs).\"\r\n },\r\n {\r\n \"Name\": \"help [command]\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/task.yaml index dbe07e0bde..28f2786c3a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-rpa-selector-healing-disabled/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/README.md new file mode 100644 index 0000000000..bd2068be6d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/README.md @@ -0,0 +1,40 @@ +# Uia Silent Noop Verify Missing — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** The "Click 'I'm Feeling Lucky'" activity in Main.xaml was authored with `InteractionMode=Simulate` and a target-less (inert) Verify Execution. Under Simulate, the click reports Successful as soon as the element is found and the event is posted — it never validates that the click had any effect — and the inert Verify (`Mode=Appears` with no verification target and empty Retry/Timeout) had nothing to check. So when the click did not take effect, no fault surfaced: the job ended Successful, and Healing Agent — which only engages on a faulting/timing-out modern UI activity — had nothing to recover. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Validation + +Passing run: `experiments/default.yaml`, model `claude-sonnet-4-6`, `--repeats 3 --max-parallel 3`. All 3 reps SUCCESS — `skill_triggered` 1.0, `file_exists` 1.0, `llm_judge` 1.0 / 0.9 / 0.8 (gate respected; no `Main.xaml` edit). Requires `SKILLS_REPO_PATH` and a tilde-free `TMPDIR/TEMP/TMP` (e.g. `C:/cetmp`) on Windows. + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name uia-silent-noop-verify-missing --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/02d09b1d04.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/02d09b1d04.json deleted file mode 100644 index ed97d539c0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/02d09b1d04.json +++ /dev/null @@ -1 +0,0 @@ ---- diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/1261b3a74b.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/1261b3a74b.json deleted file mode 100644 index f8be264685..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/1261b3a74b.json +++ /dev/null @@ -1,15 +0,0 @@ -logs exit=0 -=== traces === -traces exit=1 -=== healing-data === -healing exit=0 -{ - "Result": "Success", - "Code": "JobHealingData", - "Data": { - "JobKey": "f215a846-a117-4c56-be2a-28a2d6894cac", - "FileName": ".local/investigations/raw/triage-healing-data.zip", - "Size": 22, - "ExportId": 11702 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/2a4d4aba31.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/2a4d4aba31.json deleted file mode 100644 index b4ace67c2b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/2a4d4aba31.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f215a846-a117-4c56-be2a-28a2d6894cac", - "StartTime": "2026-06-02T09:41:25.187Z", - "EndTime": "2026-06-02T09:41:53.797Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "f215a846-a117-4c56-be2a-28a2d6894cac", - "Info": "Job completed", - "CreationTime": "2026-06-02T09:41:25.187Z", - "ReleaseName": "BlankProcess2", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20708856, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-02T09:41:53.797Z", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 65787958 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/918856e625.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/918856e625.json deleted file mode 100644 index a9c6507379..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/918856e625.json +++ /dev/null @@ -1,2 +0,0 @@ -exit=0 ---- stderr --- \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/README.md deleted file mode 100644 index 305690bbde..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Uia Silent Noop Verify Missing — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** The "Click 'I'm Feeling Lucky'" activity in Main.xaml was authored with `InteractionMode=Simulate` and a target-less (inert) Verify Execution. Under Simulate, the click reports Successful as soon as the element is found and the event is posted — it never validates that the click had any effect — and the inert Verify (`Mode=Appears` with no verification target and empty Retry/Timeout) had nothing to check. So when the click did not take effect, no fault surfaced: the job ended Successful, and Healing Agent — which only engages on a faulting/timing-out modern UI activity — had nothing to recover. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Validation - -Passing run: `experiments/default.yaml`, model `claude-sonnet-4-6`, `--repeats 3 --max-parallel 3`. All 3 reps SUCCESS — `skill_triggered` 1.0, `file_exists` 1.0, `llm_judge` 1.0 / 0.9 / 0.8 (gate respected; no `Main.xaml` edit). Requires `SKILLS_REPO_PATH` and a tilde-free `TMPDIR/TEMP/TMP` (e.g. `C:/cetmp`) on Windows. - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name uia-silent-noop-verify-missing --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/a2e52f9a0e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/a2e52f9a0e.json deleted file mode 100644 index a0589726f1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/a2e52f9a0e.json +++ /dev/null @@ -1,10 +0,0 @@ -exit=0 -error-count= 0 ---- scan all logs for license/selector/heal keywords --- -No available license: false -Agentic units: false -SelectorNotFound: false -NodeNotFound: false -UiElementNotFound: false -recovery: false -Healing agent analysis: false \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/ce7cc34810.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/ce7cc34810.json deleted file mode 100644 index 31dcfe6b01..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/ce7cc34810.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHealingData", - "Data": { - "JobKey": "f215a846-a117-4c56-be2a-28a2d6894cac", - "FileName": "", - "Size": 22, - "ExportId": 11702 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/f53ec1353f.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/f53ec1353f.json deleted file mode 100644 index b58ab62553..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/f53ec1353f.json +++ /dev/null @@ -1,432 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f215a846-a117-4c56-be2a-28a2d6894cac", - "State": "Successful", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T09:41:25.187Z", - "EndTime": "2026-06-02T09:41:53.797Z", - "CreationTime": "2026-06-02T09:41:25.187Z" - }, - { - "Key": "27cecb5c-74c6-4abb-b488-714123d9efc2", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T07:30:38.093Z", - "EndTime": "2026-06-02T07:31:37.367Z", - "CreationTime": "2026-06-02T07:30:38.093Z" - }, - { - "Key": "68d80da7-719f-44cd-b7d6-94be679f55d4", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:52:06.580Z", - "EndTime": "2026-06-02T06:52:40.293Z", - "CreationTime": "2026-06-02T06:52:06.580Z" - }, - { - "Key": "3975787c-87d9-4927-bc7c-a19165e06035", - "State": "Faulted", - "ProcessName": "BlankProcess2", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-02T06:42:21.737Z", - "EndTime": "2026-06-02T06:42:56.063Z", - "CreationTime": "2026-06-02T06:42:21.737Z" - }, - { - "Key": "b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d", - "State": "Pending", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-29T12:28:56.660Z" - }, - { - "Key": "ccc3e841-bdcf-4501-9d30-f112f098f630", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T13:09:18.353Z", - "EndTime": "2026-05-28T13:10:10.310Z", - "CreationTime": "2026-05-28T12:48:07.547Z" - }, - { - "Key": "cb00355c-3835-4c18-b37e-af5c7609c599", - "State": "Successful", - "ProcessName": "NewBlankTask", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T10:28:07.327Z", - "EndTime": "2026-05-28T10:29:16.540Z", - "CreationTime": "2026-05-28T08:48:18.580Z" - }, - { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T08:00:50.870Z", - "EndTime": "2026-05-28T08:01:02.417Z", - "CreationTime": "2026-05-27T16:24:24.603Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-26T08:48:20.617Z", - "EndTime": "2026-05-26T08:48:20.617Z", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 30, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/manifest.json deleted file mode 100644 index 577d96cab6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/m/r/manifest.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "version": 2, - "_doc": "Generated by generate_scenario.py, then hardened: docsai ask -> passthrough (query strings vary per run); added `or jobs healing-data` empty 22-byte archive rule.", - "rules": [ - { - "match": "", - "file": "02d09b1d04.json" - }, - { - "match": "or folders list --output json", - "file": "918856e625.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --output json", - "file": "f53ec1353f.json" - }, - { - "match": "or jobs get f215a846-a117-4c56-be2a-28a2d6894cac --output json", - "file": "2a4d4aba31.json" - }, - { - "match": "or jobs healing-data f215a846-a117-4c56-be2a-28a2d6894cac", - "file": "ce7cc34810.json", - "_doc": "Empty 22-byte archive — HA produced no recovery data (no fault to act on). Dispatcher returns this stdout; the -o zip file is not written, the agent reads Size:22 from stdout." - }, - { - "match": "or jobs logs f215a846-a117-4c56-be2a-28a2d6894cac --level Error --output json", - "file": "a2e52f9a0e.json" - }, - { - "match": "or jobs logs f215a846-a117-4c56-be2a-28a2d6894cac --output json", - "file": "1261b3a74b.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/uip-fixture.json new file mode 100644 index 0000000000..d78a855e8a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/data/uip-fixture.json @@ -0,0 +1,87 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "f215a846-a117-4c56-be2a-28a2d6894cac", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "exit=0\r\nerror-count= 0\r\n--- scan all logs for license/selector/heal keywords ---\r\nNo available license: false\r\nAgentic units: false\r\nSelectorNotFound: false\r\nNodeNotFound: false\r\nUiElementNotFound: false\r\nrecovery: false\r\nHealing agent analysis: false" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T09:41:25.187Z\",\r\n \"EndTime\": \"2026-06-02T09:41:53.797Z\",\r\n \"CreationTime\": \"2026-06-02T09:41:25.187Z\"\r\n },\r\n {\r\n \"Key\": \"27cecb5c-74c6-4abb-b488-714123d9efc2\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T07:30:38.093Z\",\r\n \"EndTime\": \"2026-06-02T07:31:37.367Z\",\r\n \"CreationTime\": \"2026-06-02T07:30:38.093Z\"\r\n },\r\n {\r\n \"Key\": \"68d80da7-719f-44cd-b7d6-94be679f55d4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:52:06.580Z\",\r\n \"EndTime\": \"2026-06-02T06:52:40.293Z\",\r\n \"CreationTime\": \"2026-06-02T06:52:06.580Z\"\r\n },\r\n {\r\n \"Key\": \"3975787c-87d9-4927-bc7c-a19165e06035\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess2\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-02T06:42:21.737Z\",\r\n \"EndTime\": \"2026-06-02T06:42:56.063Z\",\r\n \"CreationTime\": \"2026-06-02T06:42:21.737Z\"\r\n },\r\n {\r\n \"Key\": \"b8afdfcd-0b65-45b6-b2d7-0e0b9bb01d6d\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-29T12:28:56.660Z\"\r\n },\r\n {\r\n \"Key\": \"ccc3e841-bdcf-4501-9d30-f112f098f630\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T13:09:18.353Z\",\r\n \"EndTime\": \"2026-05-28T13:10:10.310Z\",\r\n \"CreationTime\": \"2026-05-28T12:48:07.547Z\"\r\n },\r\n {\r\n \"Key\": \"cb00355c-3835-4c18-b37e-af5c7609c599\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NewBlankTask\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T10:28:07.327Z\",\r\n \"EndTime\": \"2026-05-28T10:29:16.540Z\",\r\n \"CreationTime\": \"2026-05-28T08:48:18.580Z\"\r\n },\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T08:00:50.870Z\",\r\n \"EndTime\": \"2026-05-28T08:01:02.417Z\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 30,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f215a846-a117-4c56-be2a-28a2d6894cac", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"StartTime\": \"2026-06-02T09:41:25.187Z\",\r\n \"EndTime\": \"2026-06-02T09:41:53.797Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-06-02T09:41:25.187Z\",\r\n \"ReleaseName\": \"BlankProcess2\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20708856,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-02T09:41:53.797Z\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 65787958\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data", + "f215a846-a117-4c56-be2a-28a2d6894cac" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHealingData\",\r\n \"Data\": {\r\n \"JobKey\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"FileName\": \"\",\r\n \"Size\": 22,\r\n \"ExportId\": 11702\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f215a846-a117-4c56-be2a-28a2d6894cac", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "logs exit=0\r\n=== traces ===\r\ntraces exit=1\r\n=== healing-data ===\r\nhealing exit=0\r\n{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHealingData\",\r\n \"Data\": {\r\n \"JobKey\": \"f215a846-a117-4c56-be2a-28a2d6894cac\",\r\n \"FileName\": \".local/investigations/raw/triage-healing-data.zip\",\r\n \"Size\": 22,\r\n \"ExportId\": 11702\r\n }\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "exit=0\r\n--- stderr ---" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/task.yaml index db01f6c45b..e84fd4498e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-silent-noop-verify-missing/task.yaml @@ -19,22 +19,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/README.md new file mode 100644 index 0000000000..9e1d915310 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/README.md @@ -0,0 +1,41 @@ +# Type Into (NTypeInto) — Method Not Found / Package Version Skew (Runtime) + +Runtime troubleshooting scenario for `UiPath.UIAutomationNext.Activities` `NTypeInto` (modern +`Type Into`). + +## What this scenario exercises + +An unattended job faults immediately with `Method not found: 'Void +UiPath.UIAutomationNext.Activities.NTypeInto...'` (a `MissingMethodException`), while the same project +runs fine in Studio Debug on the developer machine. The agent must recognize this as +**`UiPath.UIAutomation.Activities` package version skew** — the robot restored a different package +version than the project was built against, so the compiled `NTypeInto` binds to a method signature the +runtime assembly does not expose — and prescribe aligning the package version across all environments +(pin + restore/republish). It must NOT diagnose a selector/targeting failure, a timeout, or corrupt +XAML. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted VB project source: `Main.xaml` with an `NTypeInto` typing `invoiceNumber` into a Chromium portal field; `project.json` pins `UiPath.UIAutomation.Activities: [24.10.3]` | + +The diagnosis is not disclosed in any agent-visible name: the project is `InvoicePortalSubmit`, the +activity is `Enter invoice number`. The prompt states the observed symptom (the pasted error string and +"works in Debug, faults on the robot"); the cause (version skew) is not named — the agent derives it +from the `MissingMethodException` signature plus the pinned dependency. + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the `UiPath.UIAutomation.Activities` version skew between the build/debug environment + and the robot as the cause, and the fix (align/pin the package version across environments and + restore/republish) — not a selector, timeout, or corrupt-XAML misdiagnosis. + +Playbook: `references/activity-packages/ui-automation/playbooks/type-into-input-failed.md` § (B). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/3db6b96527.json deleted file mode 100644 index f2da1f9210..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/3db6b96527.json +++ /dev/null @@ -1,37 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f7a2e5c8-9d31-4b62-a074-6c1e3f9b2d58", - "StartTime": "2026-07-20T08:03:11.410Z", - "EndTime": "2026-07-20T08:03:14.980Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Job faulted", - "CreationTime": "2026-07-20T08:03:11.410Z", - "ReleaseName": "InvoicePortalSubmit", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": null, - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "c4f18b20-5d0e-4a39-b16f-3e7c9d2a5b81", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-20T08:03:14.980Z", - "JobError": { - "Type": "System.MissingMethodException", - "Message": "Enter invoice number: Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(System.Activities.InArgument`1)'.", - "RuleId": null - }, - "ErrorCode": null, - "TraceId": "f7a2e5c89d314b62a0746c1e3f9b2d58", - "Id": 67023456 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/701433db1c.json deleted file mode 100644 index 73ecdeb6d6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/701433db1c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-20T08:03:14.980Z", - "JobId": 67023456 - }, - { - "State": "Running", - "Timestamp": "2026-07-20T08:03:11.980Z", - "JobId": 67023456 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/75949d659e.json deleted file mode 100644 index d09b873cae..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/75949d659e.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f7a2e5c8-9d31-4b62-a074-6c1e3f9b2d58", - "State": "Faulted", - "ProcessName": "InvoicePortalSubmit", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-20T08:03:11.410Z", - "EndTime": "2026-07-20T08:03:14.980Z", - "CreationTime": "2026-07-20T08:03:11.410Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/README.md deleted file mode 100644 index c4a2c5d2b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Type Into (NTypeInto) — Method Not Found / Package Version Skew (Runtime) - -Runtime troubleshooting scenario for `UiPath.UIAutomationNext.Activities` `NTypeInto` (modern -`Type Into`). - -## What this scenario exercises - -An unattended job faults immediately with `Method not found: 'Void -UiPath.UIAutomationNext.Activities.NTypeInto...'` (a `MissingMethodException`), while the same project -runs fine in Studio Debug on the developer machine. The agent must recognize this as -**`UiPath.UIAutomation.Activities` package version skew** — the robot restored a different package -version than the project was built against, so the compiled `NTypeInto` binds to a method signature the -runtime assembly does not expose — and prescribe aligning the package version across all environments -(pin + restore/republish). It must NOT diagnose a selector/targeting failure, a timeout, or corrupt -XAML. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted VB project source: `Main.xaml` with an `NTypeInto` typing `invoiceNumber` into a Chromium portal field; `project.json` pins `UiPath.UIAutomation.Activities: [24.10.3]` | -| `data/m/r/` | canned `Faulted` job with `JobError.Type = System.MissingMethodException`; `job-logs-error.json` / `job-logs.json` carry the `Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(...)'` message; `docsai ask` passthrough | - -The diagnosis is not leaked in any agent-visible name: the project is `InvoicePortalSubmit`, the -activity is `Enter invoice number`. The prompt states the observed symptom (the pasted error string and -"works in Debug, faults on the robot"); the cause (version skew) is not named — the agent derives it -from the `MissingMethodException` signature plus the pinned dependency. - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the `UiPath.UIAutomation.Activities` version skew between the build/debug environment - and the robot as the cause, and the fix (align/pin the package version across environments and - restore/republish) — not a selector, timeout, or corrupt-XAML misdiagnosis. - -Playbook: `references/activity-packages/ui-automation/playbooks/type-into-input-failed.md` § (B). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/b49ae1b1b0.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/b49ae1b1b0.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/b49ae1b1b0.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/b8d18266dc.json deleted file mode 100644 index 2bf7a712ae..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-20T08:03:14.870Z", - "Message": "InvoicePortalSubmit execution ended with an error: Enter invoice number: Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(System.Activities.InArgument`1)'." - }, - { - "Level": "Information", - "TimeStamp": "2026-07-20T08:03:12.640Z", - "Message": "Invoice submission started" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-20T08:03:11.980Z", - "Message": "InvoicePortalSubmit execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/d742317695.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/d742317695.json deleted file mode 100644 index 53c1833832..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/d742317695.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-20T08:03:14.870Z", - "Message": "InvoicePortalSubmit execution ended with an error: Enter invoice number: Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(System.Activities.InArgument`1)'." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/ee2628e6b6.json deleted file mode 100644 index 45d830df08..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "c4f18b20-5d0e-4a39-b16f-3e7c9d2a5b81", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/manifest.json deleted file mode 100644 index 21d84fb847..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return a canned Faulted-job fixture whose Error log carries a `Method not found: 'Void UiPath.UIAutomationNext.Activities...'` message; any other uip call returns an empty array. Matching is plain substring (first match wins); the `--level Error` rule precedes the generic logs rule so the error-only view is served for that invocation.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs logs --level Error", - "file": "d742317695.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "or jobs traces", - "file": "b49ae1b1b0.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..d67c0628d8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/data/uip-fixture.json @@ -0,0 +1,81 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-20T08:03:14.870Z\",\r\n \"Message\": \"InvoicePortalSubmit execution ended with an error: Enter invoice number: Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(System.Activities.InArgument`1)'.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c4f18b20-5d0e-4a39-b16f-3e7c9d2a5b81\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7a2e5c8-9d31-4b62-a074-6c1e3f9b2d58\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoicePortalSubmit\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-20T08:03:11.410Z\",\r\n \"EndTime\": \"2026-07-20T08:03:14.980Z\",\r\n \"CreationTime\": \"2026-07-20T08:03:11.410Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-20T08:03:14.870Z\",\r\n \"Message\": \"InvoicePortalSubmit execution ended with an error: Enter invoice number: Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(System.Activities.InArgument`1)'.\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-20T08:03:12.640Z\",\r\n \"Message\": \"Invoice submission started\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-20T08:03:11.980Z\",\r\n \"Message\": \"InvoicePortalSubmit execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-20T08:03:14.980Z\",\r\n \"JobId\": 67023456\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-20T08:03:11.980Z\",\r\n \"JobId\": 67023456\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f7a2e5c8-9d31-4b62-a074-6c1e3f9b2d58\",\r\n \"StartTime\": \"2026-07-20T08:03:11.410Z\",\r\n \"EndTime\": \"2026-07-20T08:03:14.980Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Job faulted\",\r\n \"CreationTime\": \"2026-07-20T08:03:11.410Z\",\r\n \"ReleaseName\": \"InvoicePortalSubmit\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": null,\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"c4f18b20-5d0e-4a39-b16f-3e7c9d2a5b81\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-20T08:03:14.980Z\",\r\n \"JobError\": {\r\n \"Type\": \"System.MissingMethodException\",\r\n \"Message\": \"Enter invoice number: Method not found: 'Void UiPath.UIAutomationNext.Activities.NTypeInto.set_DelayBetweenKeys(System.Activities.InArgument`1)'.\",\r\n \"RuleId\": null\r\n },\r\n \"ErrorCode\": null,\r\n \"TraceId\": \"f7a2e5c89d314b62a0746c1e3f9b2d58\",\r\n \"Id\": 67023456\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/task.yaml index e319cb3200..b238942454 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-method-not-found/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/README.md new file mode 100644 index 0000000000..6f6268a0aa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/README.md @@ -0,0 +1,39 @@ +# Type Into (NTypeInto) — Scrambled / Dropped Text (Runtime) + +Runtime troubleshooting scenario for `UiPath.UIAutomationNext.Activities` `NTypeInto` (modern +`Type Into`). + +## What this scenario exercises + +A scheduled unattended job ends `Successful`, but the account number it entered into a web portal came +out corrupted — characters missing and out of order. There is **no exception and no Error log**; the +fault is silent text corruption. The agent must recognize that `NTypeInto` ran with +`InteractionMode = HardwareEvents` and `DelayBetweenKeys = 0` and typed faster than the field accepted, +dropping/reordering keystrokes — and prescribe raising `DelayBetweenKeys` or switching the input method +to `Simulate` / `ChromiumAPI`. It must NOT diagnose a selector/targeting failure, a timeout, or a pure +silent no-op (text DID land, just wrong). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted VB project source: `Main.xaml` with an `NTypeInto` (`HardwareEvents`, `DelayBetweenKeys=0`) typing `accountNumber` into a Chromium portal field, followed by an `NGetText` read-back and Log Messages | + +The diagnosis is not disclosed in any agent-visible name: the project is `AccountPortalEntry`, the activity +is `Enter account number`, and the prompt states only the symptom (entered value wrong). The cause lives +only in the `NTypeInto` properties and the corrupted read-back value. + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the `HardwareEvents` + zero `DelayBetweenKeys` keystroke race as the cause of the + corrupted value, and the fix (increase `DelayBetweenKeys` / switch input method to + `Simulate`/`ChromiumAPI`) — not a selector, timeout, or no-op misdiagnosis. + +Playbook: `references/activity-packages/ui-automation/playbooks/type-into-input-failed.md` § (A). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/3db6b96527.json deleted file mode 100644 index fae331ae8e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/3db6b96527.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e3f9c1a4-6b28-4d70-9a15-7c0e2b8d4f61", - "StartTime": "2026-07-20T02:14:05.220Z", - "EndTime": "2026-07-20T02:14:31.640Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Job completed", - "CreationTime": "2026-07-20T02:14:05.220Z", - "ReleaseName": "AccountPortalEntry", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "b7e42a10-3c9d-4f18-a05e-2d6b8c1f4a90", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-20T02:14:31.640Z", - "JobError": {}, - "ErrorCode": null, - "TraceId": "e3f9c1a46b284d709a157c0e2b8d4f61", - "Id": 67012345 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/701433db1c.json deleted file mode 100644 index fc49d5ec9b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/701433db1c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Successful", - "Timestamp": "2026-07-20T02:14:31.640Z", - "JobId": 67012345 - }, - { - "State": "Running", - "Timestamp": "2026-07-20T02:14:05.870Z", - "JobId": 67012345 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/75949d659e.json deleted file mode 100644 index a9b60f44b7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/75949d659e.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e3f9c1a4-6b28-4d70-9a15-7c0e2b8d4f61", - "State": "Successful", - "ProcessName": "AccountPortalEntry", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-20T02:14:05.220Z", - "EndTime": "2026-07-20T02:14:31.640Z", - "CreationTime": "2026-07-20T02:14:05.220Z" - }, - { - "Key": "1a2b3c4d-5e6f-4071-8293-a4b5c6d7e8f9", - "State": "Successful", - "ProcessName": "AccountPortalEntry", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-19T02:14:04.110Z", - "EndTime": "2026-07-19T02:14:29.980Z", - "CreationTime": "2026-07-19T02:14:04.110Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/README.md deleted file mode 100644 index ffb565101d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# Type Into (NTypeInto) — Scrambled / Dropped Text (Runtime) - -Runtime troubleshooting scenario for `UiPath.UIAutomationNext.Activities` `NTypeInto` (modern -`Type Into`). - -## What this scenario exercises - -A scheduled unattended job ends `Successful`, but the account number it entered into a web portal came -out corrupted — characters missing and out of order. There is **no exception and no Error log**; the -fault is silent text corruption. The agent must recognize that `NTypeInto` ran with -`InteractionMode = HardwareEvents` and `DelayBetweenKeys = 0` and typed faster than the field accepted, -dropping/reordering keystrokes — and prescribe raising `DelayBetweenKeys` or switching the input method -to `Simulate` / `ChromiumAPI`. It must NOT diagnose a selector/targeting failure, a timeout, or a pure -silent no-op (text DID land, just wrong). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted VB project source: `Main.xaml` with an `NTypeInto` (`HardwareEvents`, `DelayBetweenKeys=0`) typing `accountNumber` into a Chromium portal field, followed by an `NGetText` read-back and Log Messages | -| `data/m/r/` | canned `Successful` job + folders/list/history; `job-logs.json` shows source `4021-7789-3316` → confirmed `4201-7789-316` (transposition + dropped char); `docsai ask` passthrough | - -The diagnosis is not leaked in any agent-visible name: the project is `AccountPortalEntry`, the activity -is `Enter account number`, and the prompt states only the symptom (entered value wrong). The cause lives -only in the `NTypeInto` properties and the corrupted read-back value. - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the `HardwareEvents` + zero `DelayBetweenKeys` keystroke race as the cause of the - corrupted value, and the fix (increase `DelayBetweenKeys` / switch input method to - `Simulate`/`ChromiumAPI`) — not a selector, timeout, or no-op misdiagnosis. - -Playbook: `references/activity-packages/ui-automation/playbooks/type-into-input-failed.md` § (A). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/b49ae1b1b0.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/b49ae1b1b0.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/b49ae1b1b0.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/b8d18266dc.json deleted file mode 100644 index bfe9e132fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-07-20T02:14:31.480Z", - "Message": "Value confirmed in portal field: 4201-7789-316" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-20T02:14:12.360Z", - "Message": "Account number read from source: 4021-7789-3316" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-20T02:14:06.010Z", - "Message": "Customer record update started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/ee2628e6b6.json deleted file mode 100644 index 235c614fc7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "b7e42a10-3c9d-4f18-a05e-2d6b8c1f4a90", - "Name": "Finance", - "Path": "Finance", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/manifest.json deleted file mode 100644 index 7ecf69be1d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return a canned Successful-job fixture (the fault is silent text corruption, not an exception); any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "or jobs traces", - "file": "b49ae1b1b0.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/uip-fixture.json new file mode 100644 index 0000000000..01ab863dba --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b7e42a10-3c9d-4f18-a05e-2d6b8c1f4a90\",\r\n \"Name\": \"Finance\",\r\n \"Path\": \"Finance\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e3f9c1a4-6b28-4d70-9a15-7c0e2b8d4f61\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"AccountPortalEntry\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-20T02:14:05.220Z\",\r\n \"EndTime\": \"2026-07-20T02:14:31.640Z\",\r\n \"CreationTime\": \"2026-07-20T02:14:05.220Z\"\r\n },\r\n {\r\n \"Key\": \"1a2b3c4d-5e6f-4071-8293-a4b5c6d7e8f9\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"AccountPortalEntry\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-19T02:14:04.110Z\",\r\n \"EndTime\": \"2026-07-19T02:14:29.980Z\",\r\n \"CreationTime\": \"2026-07-19T02:14:04.110Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-20T02:14:31.480Z\",\r\n \"Message\": \"Value confirmed in portal field: 4201-7789-316\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-20T02:14:12.360Z\",\r\n \"Message\": \"Account number read from source: 4021-7789-3316\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-20T02:14:06.010Z\",\r\n \"Message\": \"Customer record update started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Successful\",\r\n \"Timestamp\": \"2026-07-20T02:14:31.640Z\",\r\n \"JobId\": 67012345\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-20T02:14:05.870Z\",\r\n \"JobId\": 67012345\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e3f9c1a4-6b28-4d70-9a15-7c0e2b8d4f61\",\r\n \"StartTime\": \"2026-07-20T02:14:05.220Z\",\r\n \"EndTime\": \"2026-07-20T02:14:31.640Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-07-20T02:14:05.220Z\",\r\n \"ReleaseName\": \"AccountPortalEntry\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"b7e42a10-3c9d-4f18-a05e-2d6b8c1f4a90\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-20T02:14:31.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": null,\r\n \"TraceId\": \"e3f9c1a46b284d709a157c0e2b8d4f61\",\r\n \"Id\": 67012345\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/task.yaml index c70614a8cd..1b8a9444cd 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-type-into-scrambled-text/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/README.md new file mode 100644 index 0000000000..3343080a25 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/README.md @@ -0,0 +1,36 @@ +# Uia Verify Execution Failure — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +_Add a 1–3 sentence summary of the original investigation here._ + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name uia-verify-execution-failure --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/966573a66a.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/966573a66a.json deleted file mode 100644 index 906e0034d3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/966573a66a.json +++ /dev/null @@ -1,110 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 7, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/README.md deleted file mode 100644 index dba7ef5152..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Uia Verify Execution Failure — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -_Add a 1–3 sentence summary of the original investigation here._ - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name uia-verify-execution-failure --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/a06bae6235.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/a06bae6235.json deleted file mode 100644 index a3b6a061be..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/a06bae6235.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "Info": "The element was found but the verification failed because the action did not have the expected outcome.\r\n\r\nin ClickCase.xaml\r\n at NClick \"Click 'Simt c\u0103 am noroc(1)'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"ClickCase\"\r\n at ClickCase \"ClickCase\"\r\n\r\nUiPath.UIAutomationNext.Exceptions.VerifyActivityExecutionException: The element was found but the verification failed because the action did not have the expected outcome.\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyInternalAsync(IVerifyExecutionActivity activity, IRuntimeContext context, IVerifyExecutionRuntimeContext verifyContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-05-21T13:05:49.550Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 20331933, - "EntryPointPath": "ClickCase.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T13:06:01.843Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "61f7af05ad2a464da23eba2b45fb59c1", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 64782097 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/de52827b5f.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/de52827b5f.json deleted file mode 100644 index e39f3248d8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/de52827b5f.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-21T13:06:01.525Z", - "Message": "Click 'Simt c\u0103 am noroc(1)': The element was found but the verification failed because the action did not have the expected outcome.\r\nin ClickCase.xaml\r\n at NClick \"Click 'Simt c\u0103 am noroc(1)'\"\r\n at Sequence \"Do\"\r\n at NApplicationCard \"Edge Google\"\r\n at Sequence \"ClickCase\"\r\n at ClickCase \"ClickCase\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.UIAutomationNext.Exceptions.VerifyActivityExecutionException\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyInternalAsync(IVerifyExecutionActivity activity, IRuntimeContext context, IVerifyExecutionRuntimeContext verifyContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\r\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\r\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n--- End of stack trace from previous location ---\r\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\r\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/manifest.json deleted file mode 100644 index 06b01d1c41..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted --output json", - "file": "966573a66a.json" - }, - { - "match": "or jobs get 61f7af05-ad2a-464d-a23e-ba2b45fb59c1 --output json", - "file": "a06bae6235.json" - }, - { - "match": "or jobs logs 61f7af05-ad2a-464d-a23e-ba2b45fb59c1 --level Error --output json", - "file": "de52827b5f.json" - }, - { - "match": "or jobs list --folder-key 'defb8e05-e36b-4c36-bf11-0b4d08ce6cd1' --state Faulted --output json", - "file": "966573a66a.json" - }, - { - "match": "or jobs get '61f7af05-ad2a-464d-a23e-ba2b45fb59c1' --output json", - "file": "a06bae6235.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough — query strings vary between runs." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/uip-fixture.json new file mode 100644 index 0000000000..e5db5ac52d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/data/uip-fixture.json @@ -0,0 +1,65 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 7,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-21T13:06:01.525Z\",\r\n \"Message\": \"Click 'Simt c\\u0103 am noroc(1)': The element was found but the verification failed because the action did not have the expected outcome.\\r\\nin ClickCase.xaml\\r\\n at NClick \\\"Click 'Simt c\\u0103 am noroc(1)'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"ClickCase\\\"\\r\\n at ClickCase \\\"ClickCase\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.UIAutomationNext.Exceptions.VerifyActivityExecutionException\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyInternalAsync(IVerifyExecutionActivity activity, IRuntimeContext context, IVerifyExecutionRuntimeContext verifyContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"Info\": \"The element was found but the verification failed because the action did not have the expected outcome.\\r\\n\\r\\nin ClickCase.xaml\\r\\n at NClick \\\"Click 'Simt c\\u0103 am noroc(1)'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge Google\\\"\\r\\n at Sequence \\\"ClickCase\\\"\\r\\n at ClickCase \\\"ClickCase\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.VerifyActivityExecutionException: The element was found but the verification failed because the action did not have the expected outcome.\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyInternalAsync(IVerifyExecutionActivity activity, IRuntimeContext context, IVerifyExecutionRuntimeContext verifyContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.Services.VerifyExecutionService.ExecuteWithVerifyAsync(IVerifyExecutionRuntimeContext verifyContext, IRuntimeContext runtimeContext, IActivityData data, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.NClick.ExecuteAsync(IActivityContextWrapper context, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 20331933,\r\n \"EntryPointPath\": \"ClickCase.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"61f7af05ad2a464da23eba2b45fb59c1\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 64782097\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/task.yaml index 1d0dd8f19b..71b17c5400 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/uia-verify-execution-failure/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/README.md new file mode 100644 index 0000000000..a87f3edbeb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/README.md @@ -0,0 +1,9 @@ +# Scenario: web-deserialize-jsonarray-malformed + +Replays staging job `1150d469-f4dc-44fc-b39e-c89aaa5db3f9` (`ParseList`). Fault: `Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.` in Deserialize JSON Array (DeserializeJsonArray). Maps to `web-activities/playbooks/deserialize-malformed-input.md`. + +Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/1b846f8502.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/1b846f8502.json deleted file mode 100644 index 8eff748e52..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/1b846f8502.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "State": "Faulted", - "ProcessName": "ParseList", - "ReleaseName": "ParseList", - "StartTime": "2026-06-23T17:37:04.880Z", - "EndTime": "2026-06-23T17:37:06.417Z", - "CreationTime": "2026-06-23T17:36:31.630Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/44fa2ea584.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/44fa2ea584.json deleted file mode 100644 index 60e114cb67..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/44fa2ea584.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "StartTime": "2026-06-23T17:37:04.880Z", - "EndTime": "2026-06-23T17:37:06.417Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "1bed794b-f06b-4fac-8e39-629a031377f3", - "Info": "Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\r\n\r\nin Wf_ParseList.xaml\r\n at DeserializeJsonArray \"Deserialize JSON Array\"\r\n at Sequence \"Parse List\"\r\n at Wf_ParseList \"Wf_ParseList\"\r\n\r\nNewtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\r\n at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.Linq.JArray.Load(JsonReader reader, JsonLoadSettings settings)\r\n at Newtonsoft.Json.Linq.JArray.Parse(String json, JsonLoadSettings settings)\r\n at UiPath.Web.Activities.DeserializeJsonArray.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-23T17:36:31.630Z", - "ReleaseName": "ParseList", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21569943, - "EntryPointPath": "Wf_ParseList.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T17:37:06.417Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "1150d469f4dc44fcb39ec89aaa5db3f9", - "Id": 67889405 - } -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/7b2c18da98.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/7b2c18da98.json deleted file mode 100644 index b2d846d097..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/7b2c18da98.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:37:06.345Z", - "Message": "ParseList execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:37:05.110Z", - "Message": "Deserialize JSON Array: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\r\nin Wf_ParseList.xaml\r\n at DeserializeJsonArray \"Deserialize JSON Array\"\r\n at Sequence \"Parse List\"\r\n at Wf_ParseList \"Wf_ParseList\"\r\n\r\n--- .NET stack trace ---\r\nNewtonsoft.Json.JsonReaderException\r\n at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.Linq.JArray.Load(JsonReader reader, JsonLoadSettings settings)\r\n at Newtonsoft.Json.Linq.JArray.Parse(String json, JsonLoadSettings settings)\r\n at UiPath.Web.Activities.DeserializeJsonArray.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:37:05.045Z", - "Message": "ParseList execution started" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/7ee97ee9b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/7ee97ee9b6.json deleted file mode 100644 index 559f258b9c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/7ee97ee9b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-23T17:37:06.420Z", - "JobId": 67889405 - }, - { - "State": "Running", - "Timestamp": "2026-06-23T17:37:04.887Z", - "JobId": 67889405 - }, - { - "State": "Pending", - "Timestamp": "2026-06-23T17:36:31.697Z", - "JobId": 67889405 - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/9e4929c4d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/9e4929c4d3.json deleted file mode 100644 index 69c0c0e781..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/9e4929c4d3.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/README.md deleted file mode 100644 index d03716bcc1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Scenario: web-deserialize-jsonarray-malformed - -Replays staging job `1150d469-f4dc-44fc-b39e-c89aaa5db3f9` (`ParseList`). Fault: `Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.` in Deserialize JSON Array (DeserializeJsonArray). Maps to `web-activities/playbooks/deserialize-malformed-input.md`. - -Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/aa608d720c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/aa608d720c.json deleted file mode 100644 index aa54e89d7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/aa608d720c.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/bf6714348f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/bf6714348f.json deleted file mode 100644 index b59d56f49f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/bf6714348f.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/edf28a1774.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/edf28a1774.json deleted file mode 100644 index 1e9c4d68e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/edf28a1774.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:37:05.110Z", - "Message": "Deserialize JSON Array: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\r\nin Wf_ParseList.xaml\r\n at DeserializeJsonArray \"Deserialize JSON Array\"\r\n at Sequence \"Parse List\"\r\n at Wf_ParseList \"Wf_ParseList\"\r\n\r\n--- .NET stack trace ---\r\nNewtonsoft.Json.JsonReaderException\r\n at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.Linq.JArray.Load(JsonReader reader, JsonLoadSettings settings)\r\n at Newtonsoft.Json.Linq.JArray.Parse(String json, JsonLoadSettings settings)\r\n at UiPath.Web.Activities.DeserializeJsonArray.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/manifest.json deleted file mode 100644 index 201c228706..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of staging job 1150d469-f4dc-44fc-b39e-c89aaa5db3f9 (folder Shared). Real uip CLI evidence, scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 1150d469-f4dc-44fc-b39e-c89aaa5db3f9 --level Error", - "file": "edf28a1774.json" - }, - { - "match": "or jobs logs 1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "file": "7b2c18da98.json" - }, - { - "match": "or jobs get 1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "file": "44fa2ea584.json" - }, - { - "match": "or jobs history 1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "file": "7ee97ee9b6.json" - }, - { - "match": "or jobs traces 1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "file": "9e4929c4d3.json" - }, - { - "match": "traces spans get --job-key 1150d469-f4dc-44fc-b39e-c89aaa5db3f9", - "file": "bf6714348f.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "1b846f8502.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "1b846f8502.json" - }, - { - "match": "or jobs list", - "file": "1b846f8502.json" - }, - { - "match": "or folders list", - "file": "aa608d720c.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/uip-fixture.json new file mode 100644 index 0000000000..6e2bf05330 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1150d469-f4dc-44fc-b39e-c89aaa5db3f9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseList\",\r\n \"ReleaseName\": \"ParseList\",\r\n \"StartTime\": \"2026-06-23T17:37:04.880Z\",\r\n \"EndTime\": \"2026-06-23T17:37:06.417Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:31.630Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1150d469-f4dc-44fc-b39e-c89aaa5db3f9", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:37:05.110Z\",\r\n \"Message\": \"Deserialize JSON Array: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\\r\\nin Wf_ParseList.xaml\\r\\n at DeserializeJsonArray \\\"Deserialize JSON Array\\\"\\r\\n at Sequence \\\"Parse List\\\"\\r\\n at Wf_ParseList \\\"Wf_ParseList\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nNewtonsoft.Json.JsonReaderException\\r\\n at Newtonsoft.Json.JsonTextReader.ParseValue()\\r\\n at Newtonsoft.Json.Linq.JArray.Load(JsonReader reader, JsonLoadSettings settings)\\r\\n at Newtonsoft.Json.Linq.JArray.Parse(String json, JsonLoadSettings settings)\\r\\n at UiPath.Web.Activities.DeserializeJsonArray.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "1150d469-f4dc-44fc-b39e-c89aaa5db3f9" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1150d469-f4dc-44fc-b39e-c89aaa5db3f9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseList\",\r\n \"ReleaseName\": \"ParseList\",\r\n \"StartTime\": \"2026-06-23T17:37:04.880Z\",\r\n \"EndTime\": \"2026-06-23T17:37:06.417Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:31.630Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1150d469-f4dc-44fc-b39e-c89aaa5db3f9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:37:06.345Z\",\r\n \"Message\": \"ParseList execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:37:05.110Z\",\r\n \"Message\": \"Deserialize JSON Array: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\\r\\nin Wf_ParseList.xaml\\r\\n at DeserializeJsonArray \\\"Deserialize JSON Array\\\"\\r\\n at Sequence \\\"Parse List\\\"\\r\\n at Wf_ParseList \\\"Wf_ParseList\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nNewtonsoft.Json.JsonReaderException\\r\\n at Newtonsoft.Json.JsonTextReader.ParseValue()\\r\\n at Newtonsoft.Json.Linq.JArray.Load(JsonReader reader, JsonLoadSettings settings)\\r\\n at Newtonsoft.Json.Linq.JArray.Parse(String json, JsonLoadSettings settings)\\r\\n at UiPath.Web.Activities.DeserializeJsonArray.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:37:05.045Z\",\r\n \"Message\": \"ParseList execution started\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "1150d469-f4dc-44fc-b39e-c89aaa5db3f9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"1150d469-f4dc-44fc-b39e-c89aaa5db3f9\",\r\n \"StartTime\": \"2026-06-23T17:37:04.880Z\",\r\n \"EndTime\": \"2026-06-23T17:37:06.417Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"1bed794b-f06b-4fac-8e39-629a031377f3\",\r\n \"Info\": \"Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\\r\\n\\r\\nin Wf_ParseList.xaml\\r\\n at DeserializeJsonArray \\\"Deserialize JSON Array\\\"\\r\\n at Sequence \\\"Parse List\\\"\\r\\n at Wf_ParseList \\\"Wf_ParseList\\\"\\r\\n\\r\\nNewtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: n. Path '', line 0, position 0.\\r\\n at Newtonsoft.Json.JsonTextReader.ParseValue()\\r\\n at Newtonsoft.Json.Linq.JArray.Load(JsonReader reader, JsonLoadSettings settings)\\r\\n at Newtonsoft.Json.Linq.JArray.Parse(String json, JsonLoadSettings settings)\\r\\n at UiPath.Web.Activities.DeserializeJsonArray.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-23T17:36:31.630Z\",\r\n \"ReleaseName\": \"ParseList\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21569943,\r\n \"EntryPointPath\": \"Wf_ParseList.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T17:37:06.417Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"1150d469f4dc44fcb39ec89aaa5db3f9\",\r\n \"Id\": 67889405\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "1150d469-f4dc-44fc-b39e-c89aaa5db3f9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-23T17:37:06.420Z\",\r\n \"JobId\": 67889405\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-23T17:37:04.887Z\",\r\n \"JobId\": 67889405\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-23T17:36:31.697Z\",\r\n \"JobId\": 67889405\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "1150d469-f4dc-44fc-b39e-c89aaa5db3f9" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1150d469-f4dc-44fc-b39e-c89aaa5db3f9\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseList\",\r\n \"ReleaseName\": \"ParseList\",\r\n \"StartTime\": \"2026-06-23T17:37:04.880Z\",\r\n \"EndTime\": \"2026-06-23T17:37:06.417Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:31.630Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/task.yaml index 721f5ef906..047ef76c15 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-jsonarray-malformed/task.yaml @@ -20,21 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/README.md new file mode 100644 index 0000000000..77580b2717 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/README.md @@ -0,0 +1,39 @@ +# Scenario: Web Activities — Deserialize JSON malformed input + +## What this scenario uncovers + +A `ParseResponse` Orchestrator job faults in a **Deserialize JSON** activity with +`Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.` +The leading `<` means a non-JSON / HTML body was fed into the JSON parser. + +Root cause: the `DeserializeJson` activity's `JsonString` is a hardcoded non-JSON +literal (`…503 Service Unavailable…`) — there is no upstream +`HttpClient` / `NetHttpRequest` in the workflow. Maps to +`activity-packages/web-activities/playbooks/deserialize-malformed-input.md` +(literal-input branch). In production this signature means an upstream HTTP call +returned an error page / non-2xx / empty body that was parsed without a success check. + +## How the test reproduces it + + serve the recorded folder / job / logs evidence. +- **Process snapshot** (`process/WebApiClient/`) holds the failing workflow so the agent can + read the Deserialize JSON input wiring (the decisive literal-vs-HTTP evidence). +- **Fixtures** are captured from a **real staging faulted job** + (`febd892d-787b-4008-a573-4af66cf710cf`), scrubbed (host → `MOCK-HOST`, + account → `UIPATH\AUTOMATION1`). `jobs traces` returns an empty list and + `traces spans get` returns its real failure — both replayed faithfully. + +## Expected investigation chain + +`folders list` → `jobs list --state Faulted` → `jobs get ` → +`jobs logs --level Error` → read `Wf_ParseResponse.xaml` (input is a non-JSON literal, +no HTTP activity) → match `deserialize-malformed-input.md` → recommend valid-JSON / guard-the-parse fix. + +## Grading + +`skill_triggered` (uipath-troubleshoot) + `llm_judge` against `RESOLUTION.md` +(FAULT / CAUSE with a must-NOT-attribute list / RESOLUTION), final response only. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/308d7e6716.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/308d7e6716.json deleted file mode 100644 index 98ba79e254..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/308d7e6716.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:36:02.168Z", - "Message": "ParseResponse execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:36:01.717Z", - "Message": "Deserialize JSON: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\r\nin Wf_ParseResponse.xaml\r\n at DeserializeJson`1 \"Deserialize JSON\"\r\n at Sequence \"Parse Response\"\r\n at Wf_ParseResponse \"Wf_ParseResponse\"\r\n\r\n--- .NET stack trace ---\r\nNewtonsoft.Json.JsonReaderException\r\n at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\r\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)\r\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:36:01.656Z", - "Message": "ParseResponse execution started" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/64a90d375a.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/64a90d375a.json deleted file mode 100644 index c38fe74e56..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/64a90d375a.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:36:01.717Z", - "Message": "Deserialize JSON: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\r\nin Wf_ParseResponse.xaml\r\n at DeserializeJson`1 \"Deserialize JSON\"\r\n at Sequence \"Parse Response\"\r\n at Wf_ParseResponse \"Wf_ParseResponse\"\r\n\r\n--- .NET stack trace ---\r\nNewtonsoft.Json.JsonReaderException\r\n at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\r\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)\r\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/73158e111a.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/73158e111a.json deleted file mode 100644 index b4c4202c01..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/73158e111a.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "febd892d-787b-4008-a573-4af66cf710cf", - "State": "Faulted", - "ProcessName": "ParseResponse", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-23T17:36:00.477Z", - "EndTime": "2026-06-23T17:36:02.253Z", - "CreationTime": "2026-06-23T17:30:28.293Z" - } - ], - "Pagination": { - "Returned": 25, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/8fb5f28c15.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/8fb5f28c15.json deleted file mode 100644 index f53a1bf0c2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/8fb5f28c15.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-23T17:36:02.260Z", - "JobId": 67889091 - }, - { - "State": "Running", - "Timestamp": "2026-06-23T17:36:00.483Z", - "JobId": 67889091 - }, - { - "State": "Pending", - "Timestamp": "2026-06-23T17:30:28.357Z", - "JobId": 67889091 - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/9e4929c4d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/9e4929c4d3.json deleted file mode 100644 index 69c0c0e781..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/9e4929c4d3.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/README.md deleted file mode 100644 index 05d8d2cb11..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Scenario: Web Activities — Deserialize JSON malformed input - -## What this scenario uncovers - -A `ParseResponse` Orchestrator job faults in a **Deserialize JSON** activity with -`Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.` -The leading `<` means a non-JSON / HTML body was fed into the JSON parser. - -Root cause: the `DeserializeJson` activity's `JsonString` is a hardcoded non-JSON -literal (`…503 Service Unavailable…`) — there is no upstream -`HttpClient` / `NetHttpRequest` in the workflow. Maps to -`activity-packages/web-activities/playbooks/deserialize-malformed-input.md` -(literal-input branch). In production this signature means an upstream HTTP call -returned an error page / non-2xx / empty body that was parsed without a success check. - -## How the test reproduces it - -- **Mock `uip` dispatcher** (`../_shared/mock_template`) + `fixtures/mocks/responses/manifest.json` - serve the recorded folder / job / logs evidence. -- **Process snapshot** (`process/WebApiClient/`) holds the failing workflow so the agent can - read the Deserialize JSON input wiring (the decisive literal-vs-HTTP evidence). -- **Fixtures** are captured from a **real staging faulted job** - (`febd892d-787b-4008-a573-4af66cf710cf`), scrubbed (host → `MOCK-HOST`, - account → `UIPATH\AUTOMATION1`). `jobs traces` returns an empty list and - `traces spans get` returns its real failure — both replayed faithfully. - -## Expected investigation chain - -`folders list` → `jobs list --state Faulted` → `jobs get ` → -`jobs logs --level Error` → read `Wf_ParseResponse.xaml` (input is a non-JSON literal, -no HTTP activity) → match `deserialize-malformed-input.md` → recommend valid-JSON / guard-the-parse fix. - -## Grading - -`skill_triggered` (uipath-troubleshoot) + `llm_judge` against `RESOLUTION.md` -(FAULT / CAUSE with a must-NOT-attribute list / RESOLUTION), final response only. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/aa608d720c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/aa608d720c.json deleted file mode 100644 index aa54e89d7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/aa608d720c.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/bf6714348f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/bf6714348f.json deleted file mode 100644 index b59d56f49f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/bf6714348f.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/e3f1a4df3f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/e3f1a4df3f.json deleted file mode 100644 index 804eb124c0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/e3f1a4df3f.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "febd892d-787b-4008-a573-4af66cf710cf", - "StartTime": "2026-06-23T17:36:00.477Z", - "EndTime": "2026-06-23T17:36:02.253Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9cc25d1a-75fe-4a5b-9fe9-ec2fad42d8b5", - "Info": "Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\r\n\r\nin Wf_ParseResponse.xaml\r\n at DeserializeJson`1 \"Deserialize JSON\"\r\n at Sequence \"Parse Response\"\r\n at Wf_ParseResponse \"Wf_ParseResponse\"\r\n\r\nNewtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\r\n at Newtonsoft.Json.JsonTextReader.ParseValue()\r\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\r\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\r\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\r\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)\r\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-23T17:30:28.293Z", - "ReleaseName": "ParseResponse", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21569941, - "EntryPointPath": "Wf_ParseResponse.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T17:36:02.253Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "febd892d787b4008a5734af66cf710cf", - "Id": 67889091 - } -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/manifest.json deleted file mode 100644 index e02e423b8f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of staging job febd892d-787b-4008-a573-4af66cf710cf (process ParseResponse, folder Shared): Deserialize JSON received a hardcoded non-JSON HTML literal -> Newtonsoft.Json.JsonReaderException. Matches web-activities/deserialize-malformed-input.md literal-input branch (no upstream HTTP). Real uip CLI evidence, scrubbed.", - "rules": [ - { - "match": "or jobs logs febd892d-787b-4008-a573-4af66cf710cf --level Error", - "file": "64a90d375a.json" - }, - { - "match": "or jobs logs febd892d-787b-4008-a573-4af66cf710cf", - "file": "308d7e6716.json" - }, - { - "match": "or jobs get febd892d-787b-4008-a573-4af66cf710cf", - "file": "e3f1a4df3f.json" - }, - { - "match": "or jobs history febd892d-787b-4008-a573-4af66cf710cf", - "file": "8fb5f28c15.json" - }, - { - "match": "or jobs traces febd892d-787b-4008-a573-4af66cf710cf", - "file": "9e4929c4d3.json" - }, - { - "match": "traces spans get --job-key febd892d-787b-4008-a573-4af66cf710cf", - "file": "bf6714348f.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "73158e111a.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "73158e111a.json" - }, - { - "match": "or jobs list", - "file": "73158e111a.json" - }, - { - "match": "or folders list", - "file": "aa608d720c.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/uip-fixture.json new file mode 100644 index 0000000000..c78a7a4e3f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"febd892d-787b-4008-a573-4af66cf710cf\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseResponse\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-23T17:36:00.477Z\",\r\n \"EndTime\": \"2026-06-23T17:36:02.253Z\",\r\n \"CreationTime\": \"2026-06-23T17:30:28.293Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 25,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "febd892d-787b-4008-a573-4af66cf710cf", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:36:01.717Z\",\r\n \"Message\": \"Deserialize JSON: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\\r\\nin Wf_ParseResponse.xaml\\r\\n at DeserializeJson`1 \\\"Deserialize JSON\\\"\\r\\n at Sequence \\\"Parse Response\\\"\\r\\n at Wf_ParseResponse \\\"Wf_ParseResponse\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nNewtonsoft.Json.JsonReaderException\\r\\n at Newtonsoft.Json.JsonTextReader.ParseValue()\\r\\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\\r\\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\\r\\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\\r\\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\\r\\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)\\r\\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "febd892d-787b-4008-a573-4af66cf710cf" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"febd892d-787b-4008-a573-4af66cf710cf\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseResponse\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-23T17:36:00.477Z\",\r\n \"EndTime\": \"2026-06-23T17:36:02.253Z\",\r\n \"CreationTime\": \"2026-06-23T17:30:28.293Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 25,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "febd892d-787b-4008-a573-4af66cf710cf" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:36:02.168Z\",\r\n \"Message\": \"ParseResponse execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:36:01.717Z\",\r\n \"Message\": \"Deserialize JSON: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\\r\\nin Wf_ParseResponse.xaml\\r\\n at DeserializeJson`1 \\\"Deserialize JSON\\\"\\r\\n at Sequence \\\"Parse Response\\\"\\r\\n at Wf_ParseResponse \\\"Wf_ParseResponse\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nNewtonsoft.Json.JsonReaderException\\r\\n at Newtonsoft.Json.JsonTextReader.ParseValue()\\r\\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\\r\\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\\r\\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\\r\\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\\r\\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)\\r\\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:36:01.656Z\",\r\n \"Message\": \"ParseResponse execution started\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "febd892d-787b-4008-a573-4af66cf710cf" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"febd892d-787b-4008-a573-4af66cf710cf\",\r\n \"StartTime\": \"2026-06-23T17:36:00.477Z\",\r\n \"EndTime\": \"2026-06-23T17:36:02.253Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9cc25d1a-75fe-4a5b-9fe9-ec2fad42d8b5\",\r\n \"Info\": \"Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\\r\\n\\r\\nin Wf_ParseResponse.xaml\\r\\n at DeserializeJson`1 \\\"Deserialize JSON\\\"\\r\\n at Sequence \\\"Parse Response\\\"\\r\\n at Wf_ParseResponse \\\"Wf_ParseResponse\\\"\\r\\n\\r\\nNewtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\\r\\n at Newtonsoft.Json.JsonTextReader.ParseValue()\\r\\n at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)\\r\\n at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)\\r\\n at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)\\r\\n at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)\\r\\n at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)\\r\\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-23T17:30:28.293Z\",\r\n \"ReleaseName\": \"ParseResponse\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21569941,\r\n \"EntryPointPath\": \"Wf_ParseResponse.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T17:36:02.253Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"febd892d787b4008a5734af66cf710cf\",\r\n \"Id\": 67889091\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "febd892d-787b-4008-a573-4af66cf710cf" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-23T17:36:02.260Z\",\r\n \"JobId\": 67889091\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-23T17:36:00.483Z\",\r\n \"JobId\": 67889091\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-23T17:30:28.357Z\",\r\n \"JobId\": 67889091\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "febd892d-787b-4008-a573-4af66cf710cf" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"febd892d-787b-4008-a573-4af66cf710cf\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseResponse\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-23T17:36:00.477Z\",\r\n \"EndTime\": \"2026-06-23T17:36:02.253Z\",\r\n \"CreationTime\": \"2026-06-23T17:30:28.293Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 25,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/task.yaml index ed5e6967d4..6ae9354743 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-malformed-input/task.yaml @@ -26,21 +26,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/README.md new file mode 100644 index 0000000000..88d5c3a595 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/README.md @@ -0,0 +1,9 @@ +# Scenario: web-deserialize-null-input + +Replays staging job `1dc1992a-7f84-40bf-b61d-75b50f861170` (`ReadConfig`). Fault: `System.ArgumentNullException: Value cannot be null. (Parameter 'JSON string')` in Deserialize JSON (DeserializeJson). Maps to `web-activities/playbooks/deserialize-null-input.md`. + +Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/6659c514f9.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/6659c514f9.json deleted file mode 100644 index 1af786add0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/6659c514f9.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-23T17:38:06.670Z", - "JobId": 67889408 - }, - { - "State": "Running", - "Timestamp": "2026-06-23T17:38:05.187Z", - "JobId": 67889408 - }, - { - "State": "Pending", - "Timestamp": "2026-06-23T17:36:34.480Z", - "JobId": 67889408 - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/8998f26b76.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/8998f26b76.json deleted file mode 100644 index 339af271aa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/8998f26b76.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:38:06.594Z", - "Message": "ReadConfig execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:38:05.423Z", - "Message": "Deserialize JSON: Value cannot be null. (Parameter 'JSON string')\r\nin Wf_ReadConfig.xaml\r\n at DeserializeJson`1 \"Deserialize JSON\"\r\n at Sequence \"Read Config\"\r\n at Wf_ReadConfig \"Wf_ReadConfig\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentNullException\r\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:38:05.360Z", - "Message": "ReadConfig execution started" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/9e4929c4d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/9e4929c4d3.json deleted file mode 100644 index 69c0c0e781..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/9e4929c4d3.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/README.md deleted file mode 100644 index 0884b19923..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Scenario: web-deserialize-null-input - -Replays staging job `1dc1992a-7f84-40bf-b61d-75b50f861170` (`ReadConfig`). Fault: `System.ArgumentNullException: Value cannot be null. (Parameter 'JSON string')` in Deserialize JSON (DeserializeJson). Maps to `web-activities/playbooks/deserialize-null-input.md`. - -Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/aa608d720c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/aa608d720c.json deleted file mode 100644 index aa54e89d7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/aa608d720c.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/af103b1fe8.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/af103b1fe8.json deleted file mode 100644 index 59f3c97785..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/af103b1fe8.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "1dc1992a-7f84-40bf-b61d-75b50f861170", - "State": "Faulted", - "ProcessName": "ReadConfig", - "ReleaseName": "ReadConfig", - "StartTime": "2026-06-23T17:38:05.180Z", - "EndTime": "2026-06-23T17:38:06.663Z", - "CreationTime": "2026-06-23T17:36:34.407Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/b21f8593a8.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/b21f8593a8.json deleted file mode 100644 index cb0e2f4c26..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/b21f8593a8.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:38:05.423Z", - "Message": "Deserialize JSON: Value cannot be null. (Parameter 'JSON string')\r\nin Wf_ReadConfig.xaml\r\n at DeserializeJson`1 \"Deserialize JSON\"\r\n at Sequence \"Read Config\"\r\n at Wf_ReadConfig \"Wf_ReadConfig\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentNullException\r\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/bf6714348f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/bf6714348f.json deleted file mode 100644 index b59d56f49f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/bf6714348f.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/f6f01077d7.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/f6f01077d7.json deleted file mode 100644 index 7d358b9efc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/f6f01077d7.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "1dc1992a-7f84-40bf-b61d-75b50f861170", - "StartTime": "2026-06-23T17:38:05.180Z", - "EndTime": "2026-06-23T17:38:06.663Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "ddd7a9ff-e73e-43df-989e-85a00b69fb6e", - "Info": "Value cannot be null. (Parameter 'JSON string')\r\n\r\nin Wf_ReadConfig.xaml\r\n at DeserializeJson`1 \"Deserialize JSON\"\r\n at Sequence \"Read Config\"\r\n at Wf_ReadConfig \"Wf_ReadConfig\"\r\n\r\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'JSON string')\r\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-23T17:36:34.407Z", - "ReleaseName": "ReadConfig", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21569945, - "EntryPointPath": "Wf_ReadConfig.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T17:38:06.663Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "1dc1992a7f8440bfb61d75b50f861170", - "Id": 67889408 - } -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/manifest.json deleted file mode 100644 index db91dc66c4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of staging job 1dc1992a-7f84-40bf-b61d-75b50f861170 (folder Shared). Real uip CLI evidence, scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 1dc1992a-7f84-40bf-b61d-75b50f861170 --level Error", - "file": "b21f8593a8.json" - }, - { - "match": "or jobs logs 1dc1992a-7f84-40bf-b61d-75b50f861170", - "file": "8998f26b76.json" - }, - { - "match": "or jobs get 1dc1992a-7f84-40bf-b61d-75b50f861170", - "file": "f6f01077d7.json" - }, - { - "match": "or jobs history 1dc1992a-7f84-40bf-b61d-75b50f861170", - "file": "6659c514f9.json" - }, - { - "match": "or jobs traces 1dc1992a-7f84-40bf-b61d-75b50f861170", - "file": "9e4929c4d3.json" - }, - { - "match": "traces spans get --job-key 1dc1992a-7f84-40bf-b61d-75b50f861170", - "file": "bf6714348f.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "af103b1fe8.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "af103b1fe8.json" - }, - { - "match": "or jobs list", - "file": "af103b1fe8.json" - }, - { - "match": "or folders list", - "file": "aa608d720c.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/uip-fixture.json new file mode 100644 index 0000000000..7af2734a51 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1dc1992a-7f84-40bf-b61d-75b50f861170\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReadConfig\",\r\n \"ReleaseName\": \"ReadConfig\",\r\n \"StartTime\": \"2026-06-23T17:38:05.180Z\",\r\n \"EndTime\": \"2026-06-23T17:38:06.663Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:34.407Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1dc1992a-7f84-40bf-b61d-75b50f861170", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:38:05.423Z\",\r\n \"Message\": \"Deserialize JSON: Value cannot be null. (Parameter 'JSON string')\\r\\nin Wf_ReadConfig.xaml\\r\\n at DeserializeJson`1 \\\"Deserialize JSON\\\"\\r\\n at Sequence \\\"Read Config\\\"\\r\\n at Wf_ReadConfig \\\"Wf_ReadConfig\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "1dc1992a-7f84-40bf-b61d-75b50f861170" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1dc1992a-7f84-40bf-b61d-75b50f861170\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReadConfig\",\r\n \"ReleaseName\": \"ReadConfig\",\r\n \"StartTime\": \"2026-06-23T17:38:05.180Z\",\r\n \"EndTime\": \"2026-06-23T17:38:06.663Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:34.407Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1dc1992a-7f84-40bf-b61d-75b50f861170" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:38:06.594Z\",\r\n \"Message\": \"ReadConfig execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:38:05.423Z\",\r\n \"Message\": \"Deserialize JSON: Value cannot be null. (Parameter 'JSON string')\\r\\nin Wf_ReadConfig.xaml\\r\\n at DeserializeJson`1 \\\"Deserialize JSON\\\"\\r\\n at Sequence \\\"Read Config\\\"\\r\\n at Wf_ReadConfig \\\"Wf_ReadConfig\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:38:05.360Z\",\r\n \"Message\": \"ReadConfig execution started\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "1dc1992a-7f84-40bf-b61d-75b50f861170" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"1dc1992a-7f84-40bf-b61d-75b50f861170\",\r\n \"StartTime\": \"2026-06-23T17:38:05.180Z\",\r\n \"EndTime\": \"2026-06-23T17:38:06.663Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"ddd7a9ff-e73e-43df-989e-85a00b69fb6e\",\r\n \"Info\": \"Value cannot be null. (Parameter 'JSON string')\\r\\n\\r\\nin Wf_ReadConfig.xaml\\r\\n at DeserializeJson`1 \\\"Deserialize JSON\\\"\\r\\n at Sequence \\\"Read Config\\\"\\r\\n at Wf_ReadConfig \\\"Wf_ReadConfig\\\"\\r\\n\\r\\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'JSON string')\\r\\n at UiPath.Web.Activities.DeserializeJson`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-23T17:36:34.407Z\",\r\n \"ReleaseName\": \"ReadConfig\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21569945,\r\n \"EntryPointPath\": \"Wf_ReadConfig.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T17:38:06.663Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"1dc1992a7f8440bfb61d75b50f861170\",\r\n \"Id\": 67889408\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "1dc1992a-7f84-40bf-b61d-75b50f861170" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-23T17:38:06.670Z\",\r\n \"JobId\": 67889408\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-23T17:38:05.187Z\",\r\n \"JobId\": 67889408\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-23T17:36:34.480Z\",\r\n \"JobId\": 67889408\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "1dc1992a-7f84-40bf-b61d-75b50f861170" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1dc1992a-7f84-40bf-b61d-75b50f861170\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReadConfig\",\r\n \"ReleaseName\": \"ReadConfig\",\r\n \"StartTime\": \"2026-06-23T17:38:05.180Z\",\r\n \"EndTime\": \"2026-06-23T17:38:06.663Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:34.407Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/task.yaml index 93a10a5534..ce84161fac 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-null-input/task.yaml @@ -20,21 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/README.md new file mode 100644 index 0000000000..bd8f430d1e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/README.md @@ -0,0 +1,9 @@ +# Scenario: web-deserialize-xml-malformed + +Replays staging job `9937b232-5a94-44ab-8aee-ce6ea52d3bb4` (`ParseDocument`). Fault: `System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.` in Deserialize XML (DeserializeXml). Maps to `web-activities/playbooks/deserialize-malformed-input.md`. + +Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/123c5e2afe.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/123c5e2afe.json deleted file mode 100644 index 4febf42901..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/123c5e2afe.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-23T17:37:36.460Z", - "JobId": 67889406 - }, - { - "State": "Running", - "Timestamp": "2026-06-23T17:37:35.037Z", - "JobId": 67889406 - }, - { - "State": "Pending", - "Timestamp": "2026-06-23T17:36:33.287Z", - "JobId": 67889406 - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/9e4929c4d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/9e4929c4d3.json deleted file mode 100644 index 69c0c0e781..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/9e4929c4d3.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/README.md deleted file mode 100644 index 07bb8c9d5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Scenario: web-deserialize-xml-malformed - -Replays staging job `9937b232-5a94-44ab-8aee-ce6ea52d3bb4` (`ParseDocument`). Fault: `System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.` in Deserialize XML (DeserializeXml). Maps to `web-activities/playbooks/deserialize-malformed-input.md`. - -Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/aa608d720c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/aa608d720c.json deleted file mode 100644 index aa54e89d7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/aa608d720c.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/bf6714348f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/bf6714348f.json deleted file mode 100644 index b59d56f49f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/bf6714348f.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/c993680707.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/c993680707.json deleted file mode 100644 index c9bc8236be..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/c993680707.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:37:35.266Z", - "Message": "Deserialize XML: Data at the root level is invalid. Line 1, position 1.\r\nin Wf_ParseDocument.xaml\r\n at DeserializeXml \"Deserialize XML\"\r\n at Sequence \"Parse Document\"\r\n at Wf_ParseDocument \"Wf_ParseDocument\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Xml.XmlException\r\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)\r\n at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)\r\n at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()\r\n at System.Xml.XmlTextReaderImpl.ParseDocumentContent()\r\n at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)\r\n at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)\r\n at UiPath.Web.Activities.DeserializeXml.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/ee7c4d2d5c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/ee7c4d2d5c.json deleted file mode 100644 index c7912307f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/ee7c4d2d5c.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "State": "Faulted", - "ProcessName": "ParseDocument", - "ReleaseName": "ParseDocument", - "StartTime": "2026-06-23T17:37:35.027Z", - "EndTime": "2026-06-23T17:37:36.453Z", - "CreationTime": "2026-06-23T17:36:33.247Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/f008cf6202.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/f008cf6202.json deleted file mode 100644 index 0664127b77..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/f008cf6202.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "StartTime": "2026-06-23T17:37:35.027Z", - "EndTime": "2026-06-23T17:37:36.453Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c1281713-e159-46d5-82cc-f0d7ce1981be", - "Info": "Data at the root level is invalid. Line 1, position 1.\r\n\r\nin Wf_ParseDocument.xaml\r\n at DeserializeXml \"Deserialize XML\"\r\n at Sequence \"Parse Document\"\r\n at Wf_ParseDocument \"Wf_ParseDocument\"\r\n\r\nSystem.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.\r\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)\r\n at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)\r\n at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()\r\n at System.Xml.XmlTextReaderImpl.ParseDocumentContent()\r\n at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)\r\n at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)\r\n at UiPath.Web.Activities.DeserializeXml.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-23T17:36:33.247Z", - "ReleaseName": "ParseDocument", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21569944, - "EntryPointPath": "Wf_ParseDocument.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T17:37:36.453Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "9937b2325a9444ab8aeece6ea52d3bb4", - "Id": 67889406 - } -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/f2c096071e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/f2c096071e.json deleted file mode 100644 index 35fd54a4b7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/f2c096071e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:37:36.386Z", - "Message": "ParseDocument execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:37:35.266Z", - "Message": "Deserialize XML: Data at the root level is invalid. Line 1, position 1.\r\nin Wf_ParseDocument.xaml\r\n at DeserializeXml \"Deserialize XML\"\r\n at Sequence \"Parse Document\"\r\n at Wf_ParseDocument \"Wf_ParseDocument\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Xml.XmlException\r\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)\r\n at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)\r\n at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()\r\n at System.Xml.XmlTextReaderImpl.ParseDocumentContent()\r\n at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)\r\n at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)\r\n at UiPath.Web.Activities.DeserializeXml.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:37:35.195Z", - "Message": "ParseDocument execution started" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/manifest.json deleted file mode 100644 index fadaaa8a62..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of staging job 9937b232-5a94-44ab-8aee-ce6ea52d3bb4 (folder Shared). Real uip CLI evidence, scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 9937b232-5a94-44ab-8aee-ce6ea52d3bb4 --level Error", - "file": "c993680707.json" - }, - { - "match": "or jobs logs 9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "file": "f2c096071e.json" - }, - { - "match": "or jobs get 9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "file": "f008cf6202.json" - }, - { - "match": "or jobs history 9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "file": "123c5e2afe.json" - }, - { - "match": "or jobs traces 9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "file": "9e4929c4d3.json" - }, - { - "match": "traces spans get --job-key 9937b232-5a94-44ab-8aee-ce6ea52d3bb4", - "file": "bf6714348f.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "ee7c4d2d5c.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "ee7c4d2d5c.json" - }, - { - "match": "or jobs list", - "file": "ee7c4d2d5c.json" - }, - { - "match": "or folders list", - "file": "aa608d720c.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/uip-fixture.json new file mode 100644 index 0000000000..9d0456e7d9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9937b232-5a94-44ab-8aee-ce6ea52d3bb4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseDocument\",\r\n \"ReleaseName\": \"ParseDocument\",\r\n \"StartTime\": \"2026-06-23T17:37:35.027Z\",\r\n \"EndTime\": \"2026-06-23T17:37:36.453Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:33.247Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9937b232-5a94-44ab-8aee-ce6ea52d3bb4", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:37:35.266Z\",\r\n \"Message\": \"Deserialize XML: Data at the root level is invalid. Line 1, position 1.\\r\\nin Wf_ParseDocument.xaml\\r\\n at DeserializeXml \\\"Deserialize XML\\\"\\r\\n at Sequence \\\"Parse Document\\\"\\r\\n at Wf_ParseDocument \\\"Wf_ParseDocument\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Xml.XmlException\\r\\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)\\r\\n at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)\\r\\n at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()\\r\\n at System.Xml.XmlTextReaderImpl.ParseDocumentContent()\\r\\n at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)\\r\\n at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)\\r\\n at UiPath.Web.Activities.DeserializeXml.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "9937b232-5a94-44ab-8aee-ce6ea52d3bb4" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9937b232-5a94-44ab-8aee-ce6ea52d3bb4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseDocument\",\r\n \"ReleaseName\": \"ParseDocument\",\r\n \"StartTime\": \"2026-06-23T17:37:35.027Z\",\r\n \"EndTime\": \"2026-06-23T17:37:36.453Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:33.247Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9937b232-5a94-44ab-8aee-ce6ea52d3bb4" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:37:36.386Z\",\r\n \"Message\": \"ParseDocument execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:37:35.266Z\",\r\n \"Message\": \"Deserialize XML: Data at the root level is invalid. Line 1, position 1.\\r\\nin Wf_ParseDocument.xaml\\r\\n at DeserializeXml \\\"Deserialize XML\\\"\\r\\n at Sequence \\\"Parse Document\\\"\\r\\n at Wf_ParseDocument \\\"Wf_ParseDocument\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Xml.XmlException\\r\\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)\\r\\n at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)\\r\\n at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()\\r\\n at System.Xml.XmlTextReaderImpl.ParseDocumentContent()\\r\\n at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)\\r\\n at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)\\r\\n at UiPath.Web.Activities.DeserializeXml.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:37:35.195Z\",\r\n \"Message\": \"ParseDocument execution started\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9937b232-5a94-44ab-8aee-ce6ea52d3bb4" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9937b232-5a94-44ab-8aee-ce6ea52d3bb4\",\r\n \"StartTime\": \"2026-06-23T17:37:35.027Z\",\r\n \"EndTime\": \"2026-06-23T17:37:36.453Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c1281713-e159-46d5-82cc-f0d7ce1981be\",\r\n \"Info\": \"Data at the root level is invalid. Line 1, position 1.\\r\\n\\r\\nin Wf_ParseDocument.xaml\\r\\n at DeserializeXml \\\"Deserialize XML\\\"\\r\\n at Sequence \\\"Parse Document\\\"\\r\\n at Wf_ParseDocument \\\"Wf_ParseDocument\\\"\\r\\n\\r\\nSystem.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.\\r\\n at System.Xml.XmlTextReaderImpl.Throw(Exception e)\\r\\n at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)\\r\\n at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()\\r\\n at System.Xml.XmlTextReaderImpl.ParseDocumentContent()\\r\\n at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)\\r\\n at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)\\r\\n at UiPath.Web.Activities.DeserializeXml.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-23T17:36:33.247Z\",\r\n \"ReleaseName\": \"ParseDocument\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21569944,\r\n \"EntryPointPath\": \"Wf_ParseDocument.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T17:37:36.453Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"9937b2325a9444ab8aeece6ea52d3bb4\",\r\n \"Id\": 67889406\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "9937b232-5a94-44ab-8aee-ce6ea52d3bb4" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-23T17:37:36.460Z\",\r\n \"JobId\": 67889406\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-23T17:37:35.037Z\",\r\n \"JobId\": 67889406\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-23T17:36:33.287Z\",\r\n \"JobId\": 67889406\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "9937b232-5a94-44ab-8aee-ce6ea52d3bb4" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9937b232-5a94-44ab-8aee-ce6ea52d3bb4\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ParseDocument\",\r\n \"ReleaseName\": \"ParseDocument\",\r\n \"StartTime\": \"2026-06-23T17:37:35.027Z\",\r\n \"EndTime\": \"2026-06-23T17:37:36.453Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:33.247Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/task.yaml index e1c8ae90cc..b2016bcec5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-deserialize-xml-malformed/task.yaml @@ -20,21 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/README.md new file mode 100644 index 0000000000..33e567546b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/README.md @@ -0,0 +1,12 @@ +# Scenario: web-http-401-auth + +Replays job `9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90` (`InvoiceApiClient`). +Fault: `System.Net.WebException: The remote server returned an error: (401) Unauthorized.` in HTTP Request (HttpClient) on a POST. Maps to `web-activities/playbooks/http-request-auth-401-403.md`. + +The request reached the API and was rejected on **authentication**: `Wf_PostInvoice.xaml` sends the **bare token** as the `Authorization` header value (`Authorization = [apiToken]`) instead of the scheme-prefixed `"Bearer " + apiToken`, so the API rejects it with `401`. The fix is to format the header with its scheme (`"Bearer " + apiToken`) — NOT a transport (DNS/TLS), proxy (`407`), wrong-endpoint (`404`), or payload/media-type (`400/415`) fix. The decisive evidence is the workflow source: the `Authorization` header built from the raw token with no `Bearer ` prefix. + +Fixtures hand-authored in the faithful-replay shape (scrubbed: host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/InvoiceApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/11f937a7a8.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/11f937a7a8.json deleted file mode 100644 index 358bbe63e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/11f937a7a8.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-20T08:44:13.784Z", - "JobId": 68002233 - }, - { - "State": "Running", - "Timestamp": "2026-07-20T08:44:12.310Z", - "JobId": 68002233 - }, - { - "State": "Pending", - "Timestamp": "2026-07-20T08:44:09.140Z", - "JobId": 68002233 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/44bc28b771.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/44bc28b771.json deleted file mode 100644 index b495bc4f75..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/44bc28b771.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "State": "Faulted", - "ProcessName": "InvoiceApiClient", - "ReleaseName": "InvoiceApiClient", - "StartTime": "2026-07-20T08:44:12.300Z", - "EndTime": "2026-07-20T08:44:13.780Z", - "CreationTime": "2026-07-20T08:44:09.100Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/75949d659e.json deleted file mode 100644 index b495bc4f75..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/75949d659e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "State": "Faulted", - "ProcessName": "InvoiceApiClient", - "ReleaseName": "InvoiceApiClient", - "StartTime": "2026-07-20T08:44:12.300Z", - "EndTime": "2026-07-20T08:44:13.780Z", - "CreationTime": "2026-07-20T08:44:09.100Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/95f84294b4.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/95f84294b4.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/95f84294b4.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/README.md deleted file mode 100644 index 5bf1f24258..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Scenario: web-http-401-auth - -Replays job `9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90` (`InvoiceApiClient`). -Fault: `System.Net.WebException: The remote server returned an error: (401) Unauthorized.` in HTTP Request (HttpClient) on a POST. Maps to `web-activities/playbooks/http-request-auth-401-403.md`. - -The request reached the API and was rejected on **authentication**: `Wf_PostInvoice.xaml` sends the **bare token** as the `Authorization` header value (`Authorization = [apiToken]`) instead of the scheme-prefixed `"Bearer " + apiToken`, so the API rejects it with `401`. The fix is to format the header with its scheme (`"Bearer " + apiToken`) — NOT a transport (DNS/TLS), proxy (`407`), wrong-endpoint (`404`), or payload/media-type (`400/415`) fix. The decisive evidence is the workflow source: the `Authorization` header built from the raw token with no `Bearer ` prefix. - -Fixtures hand-authored in the faithful-replay shape (scrubbed: host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/InvoiceApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/a1c297b882.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/a1c297b882.json deleted file mode 100644 index ce082dc4fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/a1c297b882.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/d58013be1e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/d58013be1e.json deleted file mode 100644 index b471598a92..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/d58013be1e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-07-20T08:44:13.640Z", - "Message": "InvoiceApiClient execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-07-20T08:44:13.450Z", - "Message": "HTTP Request: The remote server returned an error: (401) Unauthorized.\r\nin Wf_PostInvoice.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Post Invoice\"\r\n at Wf_PostInvoice \"Wf_PostInvoice\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-07-20T08:44:12.520Z", - "Message": "InvoiceApiClient execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/d9b9bd77e6.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/d9b9bd77e6.json deleted file mode 100644 index b495bc4f75..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/d9b9bd77e6.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "State": "Faulted", - "ProcessName": "InvoiceApiClient", - "ReleaseName": "InvoiceApiClient", - "StartTime": "2026-07-20T08:44:12.300Z", - "EndTime": "2026-07-20T08:44:13.780Z", - "CreationTime": "2026-07-20T08:44:09.100Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/e48ed14d9c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/e48ed14d9c.json deleted file mode 100644 index e49c2f3763..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/e48ed14d9c.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-20T08:44:13.450Z", - "Message": "HTTP Request: The remote server returned an error: (401) Unauthorized.\r\nin Wf_PostInvoice.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Post Invoice\"\r\n at Wf_PostInvoice \"Wf_PostInvoice\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/edca15bc40.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/edca15bc40.json deleted file mode 100644 index 40d32acf13..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/edca15bc40.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "StartTime": "2026-07-20T08:44:12.300Z", - "EndTime": "2026-07-20T08:44:13.780Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "6f7a8b9c-0d1e-4f2a-8b3c-4d5e6f7a8b9c", - "Info": "The remote server returned an error: (401) Unauthorized.\r\n\r\nin Wf_PostInvoice.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Post Invoice\"\r\n at Wf_PostInvoice \"Wf_PostInvoice\"\r\n\r\nSystem.Net.WebException: The remote server returned an error: (401) Unauthorized.\r\n ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Unauthorized).\r\n at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()\r\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\r\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-07-20T08:44:09.100Z", - "ReleaseName": "InvoiceApiClient", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21850044, - "EntryPointPath": "Wf_PostInvoice.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-20T08:44:13.780Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "9d4f2a183c5b4e708a2f1b6c3e5d7a90", - "Id": 68002233 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/ee2628e6b6.json deleted file mode 100644 index d9ba89f4c1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "7a8b9c0d-1e2f-4a3b-8c4d-5e6f7a8b9c0d", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/manifest.json deleted file mode 100644 index 89bb7cb456..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-style replay of job 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90 (folder Shared). Scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90 --level Error", - "file": "e48ed14d9c.json" - }, - { - "match": "or jobs logs 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "file": "d58013be1e.json" - }, - { - "match": "or jobs get 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "file": "edca15bc40.json" - }, - { - "match": "or jobs history 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "file": "11f937a7a8.json" - }, - { - "match": "or jobs traces 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "file": "95f84294b4.json" - }, - { - "match": "traces spans get --job-key 9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", - "file": "a1c297b882.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b --state Faulted", - "file": "44bc28b771.json" - }, - { - "match": "or jobs list --folder-key e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b", - "file": "d9b9bd77e6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/uip-fixture.json new file mode 100644 index 0000000000..ecf02d0fcd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceApiClient\",\r\n \"ReleaseName\": \"InvoiceApiClient\",\r\n \"StartTime\": \"2026-07-20T08:44:12.300Z\",\r\n \"EndTime\": \"2026-07-20T08:44:13.780Z\",\r\n \"CreationTime\": \"2026-07-20T08:44:09.100Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-20T08:44:13.450Z\",\r\n \"Message\": \"HTTP Request: The remote server returned an error: (401) Unauthorized.\\r\\nin Wf_PostInvoice.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Post Invoice\\\"\\r\\n at Wf_PostInvoice \\\"Wf_PostInvoice\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceApiClient\",\r\n \"ReleaseName\": \"InvoiceApiClient\",\r\n \"StartTime\": \"2026-07-20T08:44:12.300Z\",\r\n \"EndTime\": \"2026-07-20T08:44:13.780Z\",\r\n \"CreationTime\": \"2026-07-20T08:44:09.100Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-20T08:44:13.640Z\",\r\n \"Message\": \"InvoiceApiClient execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-20T08:44:13.450Z\",\r\n \"Message\": \"HTTP Request: The remote server returned an error: (401) Unauthorized.\\r\\nin Wf_PostInvoice.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Post Invoice\\\"\\r\\n at Wf_PostInvoice \\\"Wf_PostInvoice\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-20T08:44:12.520Z\",\r\n \"Message\": \"InvoiceApiClient execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90\",\r\n \"StartTime\": \"2026-07-20T08:44:12.300Z\",\r\n \"EndTime\": \"2026-07-20T08:44:13.780Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"6f7a8b9c-0d1e-4f2a-8b3c-4d5e6f7a8b9c\",\r\n \"Info\": \"The remote server returned an error: (401) Unauthorized.\\r\\n\\r\\nin Wf_PostInvoice.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Post Invoice\\\"\\r\\n at Wf_PostInvoice \\\"Wf_PostInvoice\\\"\\r\\n\\r\\nSystem.Net.WebException: The remote server returned an error: (401) Unauthorized.\\r\\n ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Unauthorized).\\r\\n at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()\\r\\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\\r\\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-07-20T08:44:09.100Z\",\r\n \"ReleaseName\": \"InvoiceApiClient\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21850044,\r\n \"EntryPointPath\": \"Wf_PostInvoice.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-20T08:44:13.780Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"9d4f2a183c5b4e708a2f1b6c3e5d7a90\",\r\n \"Id\": 68002233\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-20T08:44:13.784Z\",\r\n \"JobId\": 68002233\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-20T08:44:12.310Z\",\r\n \"JobId\": 68002233\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-20T08:44:09.140Z\",\r\n \"JobId\": 68002233\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9d4f2a18-3c5b-4e70-8a2f-1b6c3e5d7a90\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceApiClient\",\r\n \"ReleaseName\": \"InvoiceApiClient\",\r\n \"StartTime\": \"2026-07-20T08:44:12.300Z\",\r\n \"EndTime\": \"2026-07-20T08:44:13.780Z\",\r\n \"CreationTime\": \"2026-07-20T08:44:09.100Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"7a8b9c0d-1e2f-4a3b-8c4d-5e6f7a8b9c0d\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e5f6a7b8-c9d0-4e1f-8a2b-3c4d5e6f7a8b\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/task.yaml index 013ee07d92..4a3accd4d1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-401-auth/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/README.md new file mode 100644 index 0000000000..45b1572369 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/README.md @@ -0,0 +1,12 @@ +# Scenario: web-http-415-content-type + +Replays job `3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75` (`CrmContactUpload`). +Fault: `System.Net.WebException: The remote server returned an error: (415) Unsupported Media Type.` in HTTP Request (HttpClient) on a POST. Maps to `web-activities/playbooks/http-request-content-type-rejected.md`. + +The request reached the CRM API and was rejected on **media type**: `Wf_CreateContact.xaml` sends a JSON body (`{"name":"Acme Corp",...}`) but advertises `BodyFormat="application/xml"`, so the server refuses the payload with `415`. The fix is to set `BodyFormat` / the `Content-Type` header to `application/json` to match the body — NOT an auth (401/403), wrong-endpoint (404), or transport fix. The decisive evidence is the workflow source: the JSON body paired with `application/xml`. + +Fixtures hand-authored in the faithful-replay shape (scrubbed: host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/CrmContactUpload/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/1b1bb73c81.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/1b1bb73c81.json deleted file mode 100644 index ce082dc4fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/1b1bb73c81.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/201a344869.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/201a344869.json deleted file mode 100644 index 135ada7083..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/201a344869.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-07-20T13:02:42.500Z", - "Message": "CrmContactUpload execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-07-20T13:02:42.300Z", - "Message": "HTTP Request: The remote server returned an error: (415) Unsupported Media Type.\r\nin Wf_CreateContact.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Create CRM Contact\"\r\n at Wf_CreateContact \"Wf_CreateContact\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-07-20T13:02:41.400Z", - "Message": "CrmContactUpload execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/63636db9b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/63636db9b6.json deleted file mode 100644 index 0fd5740fbf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/63636db9b6.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-20T13:02:42.644Z", - "JobId": 68005512 - }, - { - "State": "Running", - "Timestamp": "2026-07-20T13:02:41.210Z", - "JobId": 68005512 - }, - { - "State": "Pending", - "Timestamp": "2026-07-20T13:02:38.140Z", - "JobId": 68005512 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/72f9d782d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/72f9d782d3.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/72f9d782d3.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/75949d659e.json deleted file mode 100644 index 864e1a784c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/75949d659e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "State": "Faulted", - "ProcessName": "CrmContactUpload", - "ReleaseName": "CrmContactUpload", - "StartTime": "2026-07-20T13:02:41.200Z", - "EndTime": "2026-07-20T13:02:42.640Z", - "CreationTime": "2026-07-20T13:02:38.100Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/README.md deleted file mode 100644 index 01f61e254d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Scenario: web-http-415-content-type - -Replays job `3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75` (`CrmContactUpload`). -Fault: `System.Net.WebException: The remote server returned an error: (415) Unsupported Media Type.` in HTTP Request (HttpClient) on a POST. Maps to `web-activities/playbooks/http-request-content-type-rejected.md`. - -The request reached the CRM API and was rejected on **media type**: `Wf_CreateContact.xaml` sends a JSON body (`{"name":"Acme Corp",...}`) but advertises `BodyFormat="application/xml"`, so the server refuses the payload with `415`. The fix is to set `BodyFormat` / the `Content-Type` header to `application/json` to match the body — NOT an auth (401/403), wrong-endpoint (404), or transport fix. The decisive evidence is the workflow source: the JSON body paired with `application/xml`. - -Fixtures hand-authored in the faithful-replay shape (scrubbed: host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/CrmContactUpload/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/e053591c4a.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/e053591c4a.json deleted file mode 100644 index 5e772167dc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/e053591c4a.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "StartTime": "2026-07-20T13:02:41.200Z", - "EndTime": "2026-07-20T13:02:42.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "5d6e7f8a-9b0c-4d1e-8f2a-3b4c5d6e7f80", - "Info": "The remote server returned an error: (415) Unsupported Media Type.\r\n\r\nin Wf_CreateContact.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Create CRM Contact\"\r\n at Wf_CreateContact \"Wf_CreateContact\"\r\n\r\nSystem.Net.WebException: The remote server returned an error: (415) Unsupported Media Type.\r\n ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 415 (Unsupported Media Type).\r\n at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()\r\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\r\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-07-20T13:02:38.100Z", - "ReleaseName": "CrmContactUpload", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21855120, - "EntryPointPath": "Wf_CreateContact.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-20T13:02:42.640Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3f8d5a912b7c4e609a1f6c8e4d2b0a75", - "Id": 68005512 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/ed7a6ee927.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/ed7a6ee927.json deleted file mode 100644 index bb99560bff..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/ed7a6ee927.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-20T13:02:42.300Z", - "Message": "HTTP Request: The remote server returned an error: (415) Unsupported Media Type.\r\nin Wf_CreateContact.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Create CRM Contact\"\r\n at Wf_CreateContact \"Wf_CreateContact\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/ee2628e6b6.json deleted file mode 100644 index 88b01d34b0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6e7f8a9b-0c1d-4e2f-8a3b-4c5d6e7f8a9b", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/f65936d6cb.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/f65936d6cb.json deleted file mode 100644 index 864e1a784c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/f65936d6cb.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "State": "Faulted", - "ProcessName": "CrmContactUpload", - "ReleaseName": "CrmContactUpload", - "StartTime": "2026-07-20T13:02:41.200Z", - "EndTime": "2026-07-20T13:02:42.640Z", - "CreationTime": "2026-07-20T13:02:38.100Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/fba7e6dc56.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/fba7e6dc56.json deleted file mode 100644 index 864e1a784c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/fba7e6dc56.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "State": "Faulted", - "ProcessName": "CrmContactUpload", - "ReleaseName": "CrmContactUpload", - "StartTime": "2026-07-20T13:02:41.200Z", - "EndTime": "2026-07-20T13:02:42.640Z", - "CreationTime": "2026-07-20T13:02:38.100Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/manifest.json deleted file mode 100644 index 5d247f9882..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-style replay of job 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75 (folder Shared). Scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75 --level Error", - "file": "ed7a6ee927.json" - }, - { - "match": "or jobs logs 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "file": "201a344869.json" - }, - { - "match": "or jobs get 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "file": "e053591c4a.json" - }, - { - "match": "or jobs history 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "file": "63636db9b6.json" - }, - { - "match": "or jobs traces 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "file": "72f9d782d3.json" - }, - { - "match": "traces spans get --job-key 3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", - "file": "1b1bb73c81.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80 --state Faulted", - "file": "f65936d6cb.json" - }, - { - "match": "or jobs list --folder-key d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80", - "file": "fba7e6dc56.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/uip-fixture.json new file mode 100644 index 0000000000..5e388a4f90 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CrmContactUpload\",\r\n \"ReleaseName\": \"CrmContactUpload\",\r\n \"StartTime\": \"2026-07-20T13:02:41.200Z\",\r\n \"EndTime\": \"2026-07-20T13:02:42.640Z\",\r\n \"CreationTime\": \"2026-07-20T13:02:38.100Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-20T13:02:42.300Z\",\r\n \"Message\": \"HTTP Request: The remote server returned an error: (415) Unsupported Media Type.\\r\\nin Wf_CreateContact.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Create CRM Contact\\\"\\r\\n at Wf_CreateContact \\\"Wf_CreateContact\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CrmContactUpload\",\r\n \"ReleaseName\": \"CrmContactUpload\",\r\n \"StartTime\": \"2026-07-20T13:02:41.200Z\",\r\n \"EndTime\": \"2026-07-20T13:02:42.640Z\",\r\n \"CreationTime\": \"2026-07-20T13:02:38.100Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-20T13:02:42.500Z\",\r\n \"Message\": \"CrmContactUpload execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-20T13:02:42.300Z\",\r\n \"Message\": \"HTTP Request: The remote server returned an error: (415) Unsupported Media Type.\\r\\nin Wf_CreateContact.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Create CRM Contact\\\"\\r\\n at Wf_CreateContact \\\"Wf_CreateContact\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-20T13:02:41.400Z\",\r\n \"Message\": \"CrmContactUpload execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75\",\r\n \"StartTime\": \"2026-07-20T13:02:41.200Z\",\r\n \"EndTime\": \"2026-07-20T13:02:42.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"5d6e7f8a-9b0c-4d1e-8f2a-3b4c5d6e7f80\",\r\n \"Info\": \"The remote server returned an error: (415) Unsupported Media Type.\\r\\n\\r\\nin Wf_CreateContact.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Create CRM Contact\\\"\\r\\n at Wf_CreateContact \\\"Wf_CreateContact\\\"\\r\\n\\r\\nSystem.Net.WebException: The remote server returned an error: (415) Unsupported Media Type.\\r\\n ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 415 (Unsupported Media Type).\\r\\n at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()\\r\\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\\r\\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-07-20T13:02:38.100Z\",\r\n \"ReleaseName\": \"CrmContactUpload\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21855120,\r\n \"EntryPointPath\": \"Wf_CreateContact.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-20T13:02:42.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3f8d5a912b7c4e609a1f6c8e4d2b0a75\",\r\n \"Id\": 68005512\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-20T13:02:42.644Z\",\r\n \"JobId\": 68005512\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-20T13:02:41.210Z\",\r\n \"JobId\": 68005512\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-20T13:02:38.140Z\",\r\n \"JobId\": 68005512\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3f8d5a91-2b7c-4e60-9a1f-6c8e4d2b0a75\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CrmContactUpload\",\r\n \"ReleaseName\": \"CrmContactUpload\",\r\n \"StartTime\": \"2026-07-20T13:02:41.200Z\",\r\n \"EndTime\": \"2026-07-20T13:02:42.640Z\",\r\n \"CreationTime\": \"2026-07-20T13:02:38.100Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6e7f8a9b-0c1d-4e2f-8a3b-4c5d6e7f8a9b\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-4d0e-9f1a-3b4c5d6e7f80\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/task.yaml index 9720dbf336..0c71d84f4f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-415-content-type/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/README.md new file mode 100644 index 0000000000..f092dfabd8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/README.md @@ -0,0 +1,9 @@ +# Scenario: web-http-connection-failure + +Replays staging job `419a6787-6cc6-4b93-bae4-ba985bf6fe67` (`CallService`). Fault: `System.Net.WebException: No such host is known. (service-endpoint-unavailable.invalid:80)` in HTTP Request (HttpClient). Maps to `web-activities/playbooks/http-request-connection-failure.md`. + +Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/174d4cb2fe.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/174d4cb2fe.json deleted file mode 100644 index 50ddebd6f2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/174d4cb2fe.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "StartTime": "2026-06-23T17:36:33.547Z", - "EndTime": "2026-06-23T17:36:34.967Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "766a623e-ffe2-4225-94f2-8d9285f96594", - "Info": "No such host is known. (service-endpoint-unavailable.invalid:80)\r\n\r\nin Wf_CallService.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Call Service\"\r\n at Wf_CallService \"Wf_CallService\"\r\n\r\nSystem.Net.WebException: No such host is known. (service-endpoint-unavailable.invalid:80)\r\n ---> System.Net.Http.HttpRequestException: No such host is known. (service-endpoint-unavailable.invalid:80)\r\n ---> System.Net.Sockets.SocketException: No such host is known.\r\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\r\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\r\n at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\r\n at System.Net.HttpWebRequest.<>c__DisplayClass219_0.<b__1>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\r\n at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)\r\n at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\r\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\r\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-23T17:36:30.333Z", - "ReleaseName": "CallService", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21569942, - "EntryPointPath": "Wf_CallService.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T17:36:34.967Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "419a67876cc64b93bae4ba985bf6fe67", - "Id": 67889404 - } -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/24f7fc4512.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/24f7fc4512.json deleted file mode 100644 index 87bb417d81..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/24f7fc4512.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "State": "Faulted", - "ProcessName": "CallService", - "ReleaseName": "CallService", - "StartTime": "2026-06-23T17:36:33.547Z", - "EndTime": "2026-06-23T17:36:34.967Z", - "CreationTime": "2026-06-23T17:36:30.333Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/6f91432585.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/6f91432585.json deleted file mode 100644 index 8627107fe5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/6f91432585.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-23T17:36:34.970Z", - "JobId": 67889404 - }, - { - "State": "Running", - "Timestamp": "2026-06-23T17:36:33.553Z", - "JobId": 67889404 - }, - { - "State": "Pending", - "Timestamp": "2026-06-23T17:36:30.373Z", - "JobId": 67889404 - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/9e4929c4d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/9e4929c4d3.json deleted file mode 100644 index 69c0c0e781..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/9e4929c4d3.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/README.md deleted file mode 100644 index 9809a666e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Scenario: web-http-connection-failure - -Replays staging job `419a6787-6cc6-4b93-bae4-ba985bf6fe67` (`CallService`). Fault: `System.Net.WebException: No such host is known. (service-endpoint-unavailable.invalid:80)` in HTTP Request (HttpClient). Maps to `web-activities/playbooks/http-request-connection-failure.md`. - -Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/a94430b3db.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/a94430b3db.json deleted file mode 100644 index 51593a2d74..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/a94430b3db.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:36:34.901Z", - "Message": "CallService execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:36:33.952Z", - "Message": "HTTP Request: No such host is known. (service-endpoint-unavailable.invalid:80)\r\nin Wf_CallService.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Call Service\"\r\n at Wf_CallService \"Wf_CallService\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:36:33.747Z", - "Message": "CallService execution started" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/aa608d720c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/aa608d720c.json deleted file mode 100644 index aa54e89d7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/aa608d720c.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/b4a7bc912e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/b4a7bc912e.json deleted file mode 100644 index 25a21f82b0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/b4a7bc912e.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:36:33.952Z", - "Message": "HTTP Request: No such host is known. (service-endpoint-unavailable.invalid:80)\r\nin Wf_CallService.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Call Service\"\r\n at Wf_CallService \"Wf_CallService\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/bf6714348f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/bf6714348f.json deleted file mode 100644 index b59d56f49f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/bf6714348f.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/manifest.json deleted file mode 100644 index a7b0f696e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of staging job 419a6787-6cc6-4b93-bae4-ba985bf6fe67 (folder Shared). Real uip CLI evidence, scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 419a6787-6cc6-4b93-bae4-ba985bf6fe67 --level Error", - "file": "b4a7bc912e.json" - }, - { - "match": "or jobs logs 419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "file": "a94430b3db.json" - }, - { - "match": "or jobs get 419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "file": "174d4cb2fe.json" - }, - { - "match": "or jobs history 419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "file": "6f91432585.json" - }, - { - "match": "or jobs traces 419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "file": "9e4929c4d3.json" - }, - { - "match": "traces spans get --job-key 419a6787-6cc6-4b93-bae4-ba985bf6fe67", - "file": "bf6714348f.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "24f7fc4512.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "24f7fc4512.json" - }, - { - "match": "or jobs list", - "file": "24f7fc4512.json" - }, - { - "match": "or folders list", - "file": "aa608d720c.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/uip-fixture.json new file mode 100644 index 0000000000..8faab56b63 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"419a6787-6cc6-4b93-bae4-ba985bf6fe67\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CallService\",\r\n \"ReleaseName\": \"CallService\",\r\n \"StartTime\": \"2026-06-23T17:36:33.547Z\",\r\n \"EndTime\": \"2026-06-23T17:36:34.967Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:30.333Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "419a6787-6cc6-4b93-bae4-ba985bf6fe67", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:36:33.952Z\",\r\n \"Message\": \"HTTP Request: No such host is known. (service-endpoint-unavailable.invalid:80)\\r\\nin Wf_CallService.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Call Service\\\"\\r\\n at Wf_CallService \\\"Wf_CallService\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "419a6787-6cc6-4b93-bae4-ba985bf6fe67" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"419a6787-6cc6-4b93-bae4-ba985bf6fe67\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CallService\",\r\n \"ReleaseName\": \"CallService\",\r\n \"StartTime\": \"2026-06-23T17:36:33.547Z\",\r\n \"EndTime\": \"2026-06-23T17:36:34.967Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:30.333Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "419a6787-6cc6-4b93-bae4-ba985bf6fe67" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:36:34.901Z\",\r\n \"Message\": \"CallService execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:36:33.952Z\",\r\n \"Message\": \"HTTP Request: No such host is known. (service-endpoint-unavailable.invalid:80)\\r\\nin Wf_CallService.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Call Service\\\"\\r\\n at Wf_CallService \\\"Wf_CallService\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:36:33.747Z\",\r\n \"Message\": \"CallService execution started\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "419a6787-6cc6-4b93-bae4-ba985bf6fe67" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"419a6787-6cc6-4b93-bae4-ba985bf6fe67\",\r\n \"StartTime\": \"2026-06-23T17:36:33.547Z\",\r\n \"EndTime\": \"2026-06-23T17:36:34.967Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"766a623e-ffe2-4225-94f2-8d9285f96594\",\r\n \"Info\": \"No such host is known. (service-endpoint-unavailable.invalid:80)\\r\\n\\r\\nin Wf_CallService.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Call Service\\\"\\r\\n at Wf_CallService \\\"Wf_CallService\\\"\\r\\n\\r\\nSystem.Net.WebException: No such host is known. (service-endpoint-unavailable.invalid:80)\\r\\n ---> System.Net.Http.HttpRequestException: No such host is known. (service-endpoint-unavailable.invalid:80)\\r\\n ---> System.Net.Sockets.SocketException: No such host is known.\\r\\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)\\r\\n at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)\\r\\n at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)\\r\\n at System.Net.HttpWebRequest.<>c__DisplayClass219_0.<b__1>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\\r\\n at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\\r\\n at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)\\r\\n at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)\\r\\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\\r\\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-23T17:36:30.333Z\",\r\n \"ReleaseName\": \"CallService\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21569942,\r\n \"EntryPointPath\": \"Wf_CallService.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T17:36:34.967Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"419a67876cc64b93bae4ba985bf6fe67\",\r\n \"Id\": 67889404\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "419a6787-6cc6-4b93-bae4-ba985bf6fe67" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-23T17:36:34.970Z\",\r\n \"JobId\": 67889404\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-23T17:36:33.553Z\",\r\n \"JobId\": 67889404\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-23T17:36:30.373Z\",\r\n \"JobId\": 67889404\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "419a6787-6cc6-4b93-bae4-ba985bf6fe67" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"419a6787-6cc6-4b93-bae4-ba985bf6fe67\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"CallService\",\r\n \"ReleaseName\": \"CallService\",\r\n \"StartTime\": \"2026-06-23T17:36:33.547Z\",\r\n \"EndTime\": \"2026-06-23T17:36:34.967Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:30.333Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/task.yaml index b0ac546c23..4e556db94b 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-connection-failure/task.yaml @@ -20,21 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/README.md new file mode 100644 index 0000000000..884957d129 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/README.md @@ -0,0 +1,12 @@ +# Scenario: web-http-proxy-blocked + +Replays unattended job `7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40` (`PaymentGatewaySync`). +Fault: `System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.` in HTTP Request (HttpClient) on a POST to an external endpoint. Maps to `web-activities/playbooks/http-request-proxy-blocked.md`. + +The discriminating signal is **environment**: the process runs fine from Studio on the developer's laptop but faults only on the scheduled **unattended robot** on the corporate server network. The `407` means the corporate proxy is in the path but the robot's request carries no proxy credentials (the Web activities expose no proxy property and don't pick up the interactive user's system proxy under a service account). The fix is to route the robot's HTTP egress through the proxy with credentials — NOT a DNS/TLS fix, a wrong-endpoint fix, or an auth fix on the target API. + +Fixtures hand-authored in the faithful-replay shape (scrubbed: host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/PaymentGatewaySync/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/074f2ec219.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/074f2ec219.json deleted file mode 100644 index d09a59b3ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/074f2ec219.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-21T02:15:09.410Z", - "Message": "HTTP Request: The remote server returned an error: (407) Proxy Authentication Required.\r\nin Wf_PostPayment.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Post Payment\"\r\n at Wf_PostPayment \"Wf_PostPayment\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/2f1f00231d.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/2f1f00231d.json deleted file mode 100644 index ce082dc4fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/2f1f00231d.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/3ca9422402.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/3ca9422402.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/3ca9422402.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/466550c113.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/466550c113.json deleted file mode 100644 index 97228c9581..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/466550c113.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-21T02:15:09.884Z", - "JobId": 68012774 - }, - { - "State": "Running", - "Timestamp": "2026-07-21T02:15:04.130Z", - "JobId": 68012774 - }, - { - "State": "Pending", - "Timestamp": "2026-07-21T02:15:00.540Z", - "JobId": 68012774 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/4fffba3818.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/4fffba3818.json deleted file mode 100644 index 10d5a7ead8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/4fffba3818.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "State": "Faulted", - "ProcessName": "PaymentGatewaySync", - "ReleaseName": "PaymentGatewaySync", - "StartTime": "2026-07-21T02:15:04.120Z", - "EndTime": "2026-07-21T02:15:09.880Z", - "CreationTime": "2026-07-21T02:15:00.500Z", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/75949d659e.json deleted file mode 100644 index 10d5a7ead8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/75949d659e.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "State": "Faulted", - "ProcessName": "PaymentGatewaySync", - "ReleaseName": "PaymentGatewaySync", - "StartTime": "2026-07-21T02:15:04.120Z", - "EndTime": "2026-07-21T02:15:09.880Z", - "CreationTime": "2026-07-21T02:15:00.500Z", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/README.md deleted file mode 100644 index 32fe4abfe4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Scenario: web-http-proxy-blocked - -Replays unattended job `7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40` (`PaymentGatewaySync`). -Fault: `System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.` in HTTP Request (HttpClient) on a POST to an external endpoint. Maps to `web-activities/playbooks/http-request-proxy-blocked.md`. - -The discriminating signal is **environment**: the process runs fine from Studio on the developer's laptop but faults only on the scheduled **unattended robot** on the corporate server network. The `407` means the corporate proxy is in the path but the robot's request carries no proxy credentials (the Web activities expose no proxy property and don't pick up the interactive user's system proxy under a service account). The fix is to route the robot's HTTP egress through the proxy with credentials — NOT a DNS/TLS fix, a wrong-endpoint fix, or an auth fix on the target API. - -Fixtures hand-authored in the faithful-replay shape (scrubbed: host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/PaymentGatewaySync/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/dbae1d9afe.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/dbae1d9afe.json deleted file mode 100644 index 10d5a7ead8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/dbae1d9afe.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "State": "Faulted", - "ProcessName": "PaymentGatewaySync", - "ReleaseName": "PaymentGatewaySync", - "StartTime": "2026-07-21T02:15:04.120Z", - "EndTime": "2026-07-21T02:15:09.880Z", - "CreationTime": "2026-07-21T02:15:00.500Z", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/ee2628e6b6.json deleted file mode 100644 index 07d4c15ab7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "9b8a7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/f4a60a089f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/f4a60a089f.json deleted file mode 100644 index e9c29b8494..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/f4a60a089f.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "StartTime": "2026-07-21T02:15:04.120Z", - "EndTime": "2026-07-21T02:15:09.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "8c1d2e3f-aa4b-4c5d-9e6f-7a8b9c0d1e2f", - "Info": "The remote server returned an error: (407) Proxy Authentication Required.\r\n\r\nin Wf_PostPayment.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Post Payment\"\r\n at Wf_PostPayment \"Wf_PostPayment\"\r\n\r\nSystem.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.\r\n ---> System.Net.Http.HttpRequestException: The remote server returned an error: (407) Proxy Authentication Required.\r\n at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\r\n at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\r\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\r\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\r\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\r\n --- End of inner exception stack trace ---\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-07-21T02:15:00.500Z", - "ReleaseName": "PaymentGatewaySync", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21870031, - "EntryPointPath": "Wf_PostPayment.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-21T02:15:09.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "7b2e1c049f3a4d218c6e2a5f9b1d3e40", - "Id": 68012774 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/f6c37546d7.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/f6c37546d7.json deleted file mode 100644 index fa787e2428..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/f6c37546d7.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-07-21T02:15:09.700Z", - "Message": "PaymentGatewaySync execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-07-21T02:15:09.410Z", - "Message": "HTTP Request: The remote server returned an error: (407) Proxy Authentication Required.\r\nin Wf_PostPayment.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Post Payment\"\r\n at Wf_PostPayment \"Wf_PostPayment\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-07-21T02:15:04.320Z", - "Message": "PaymentGatewaySync execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/manifest.json deleted file mode 100644 index fa489369a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-style replay of unattended job 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40 (folder Shared). Scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40 --level Error", - "file": "074f2ec219.json" - }, - { - "match": "or jobs logs 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "file": "f6c37546d7.json" - }, - { - "match": "or jobs get 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "file": "f4a60a089f.json" - }, - { - "match": "or jobs history 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "file": "466550c113.json" - }, - { - "match": "or jobs traces 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "file": "3ca9422402.json" - }, - { - "match": "traces spans get --job-key 7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", - "file": "2f1f00231d.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d --state Faulted", - "file": "4fffba3818.json" - }, - { - "match": "or jobs list --folder-key c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", - "file": "dbae1d9afe.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/uip-fixture.json new file mode 100644 index 0000000000..1fdc3c2dd0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PaymentGatewaySync\",\r\n \"ReleaseName\": \"PaymentGatewaySync\",\r\n \"StartTime\": \"2026-07-21T02:15:04.120Z\",\r\n \"EndTime\": \"2026-07-21T02:15:09.880Z\",\r\n \"CreationTime\": \"2026-07-21T02:15:00.500Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-21T02:15:09.410Z\",\r\n \"Message\": \"HTTP Request: The remote server returned an error: (407) Proxy Authentication Required.\\r\\nin Wf_PostPayment.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Post Payment\\\"\\r\\n at Wf_PostPayment \\\"Wf_PostPayment\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PaymentGatewaySync\",\r\n \"ReleaseName\": \"PaymentGatewaySync\",\r\n \"StartTime\": \"2026-07-21T02:15:04.120Z\",\r\n \"EndTime\": \"2026-07-21T02:15:09.880Z\",\r\n \"CreationTime\": \"2026-07-21T02:15:00.500Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-21T02:15:09.700Z\",\r\n \"Message\": \"PaymentGatewaySync execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-21T02:15:09.410Z\",\r\n \"Message\": \"HTTP Request: The remote server returned an error: (407) Proxy Authentication Required.\\r\\nin Wf_PostPayment.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Post Payment\\\"\\r\\n at Wf_PostPayment \\\"Wf_PostPayment\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-21T02:15:04.320Z\",\r\n \"Message\": \"PaymentGatewaySync execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40\",\r\n \"StartTime\": \"2026-07-21T02:15:04.120Z\",\r\n \"EndTime\": \"2026-07-21T02:15:09.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"8c1d2e3f-aa4b-4c5d-9e6f-7a8b9c0d1e2f\",\r\n \"Info\": \"The remote server returned an error: (407) Proxy Authentication Required.\\r\\n\\r\\nin Wf_PostPayment.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Post Payment\\\"\\r\\n at Wf_PostPayment \\\"Wf_PostPayment\\\"\\r\\n\\r\\nSystem.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required.\\r\\n ---> System.Net.Http.HttpRequestException: The remote server returned an error: (407) Proxy Authentication Required.\\r\\n at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)\\r\\n at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)\\r\\n at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)\\r\\n at System.Net.HttpWebRequest.SendRequest(Boolean async)\\r\\n at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-07-21T02:15:00.500Z\",\r\n \"ReleaseName\": \"PaymentGatewaySync\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21870031,\r\n \"EntryPointPath\": \"Wf_PostPayment.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-21T02:15:09.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"7b2e1c049f3a4d218c6e2a5f9b1d3e40\",\r\n \"Id\": 68012774\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-21T02:15:09.884Z\",\r\n \"JobId\": 68012774\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-21T02:15:04.130Z\",\r\n \"JobId\": 68012774\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-21T02:15:00.540Z\",\r\n \"JobId\": 68012774\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"7b2e1c04-9f3a-4d21-8c6e-2a5f9b1d3e40\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PaymentGatewaySync\",\r\n \"ReleaseName\": \"PaymentGatewaySync\",\r\n \"StartTime\": \"2026-07-21T02:15:04.120Z\",\r\n \"EndTime\": \"2026-07-21T02:15:09.880Z\",\r\n \"CreationTime\": \"2026-07-21T02:15:00.500Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9b8a7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1a2b3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/task.yaml index 16147ab13d..bab47edcb4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-proxy-blocked/task.yaml @@ -19,21 +19,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/README.md new file mode 100644 index 0000000000..46cf2da3c4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/README.md @@ -0,0 +1,9 @@ +# Scenario: web-http-timeout + +Replays staging job `3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12` (`FetchReport`). Fault: `System.Net.WebException: The operation has timed out.` in HTTP Request (HttpClient). Maps to `web-activities/playbooks/http-request-timeout.md`. + +Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/75e53f025f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/75e53f025f.json deleted file mode 100644 index 676dff6e3f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/75e53f025f.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:36:31.667Z", - "Message": "HTTP Request: The operation has timed out.\r\nin Wf_FetchReport.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Fetch Report\"\r\n at Wf_FetchReport \"Wf_FetchReport\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/875d9f9c3a.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/875d9f9c3a.json deleted file mode 100644 index 18c12e82e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/875d9f9c3a.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-23T17:36:32.113Z", - "JobId": 67889402 - }, - { - "State": "Running", - "Timestamp": "2026-06-23T17:36:29.240Z", - "JobId": 67889402 - }, - { - "State": "Pending", - "Timestamp": "2026-06-23T17:36:28.953Z", - "JobId": 67889402 - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/9e4929c4d3.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/9e4929c4d3.json deleted file mode 100644 index 69c0c0e781..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/9e4929c4d3.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/README.md deleted file mode 100644 index 296892e8f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# Scenario: web-http-timeout - -Replays staging job `3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12` (`FetchReport`). Fault: `System.Net.WebException: The operation has timed out.` in HTTP Request (HttpClient). Maps to `web-activities/playbooks/http-request-timeout.md`. - -Fixtures captured from a real staging faulted job, scrubbed (host → MOCK-HOST, account → UIPATH\AUTOMATION1). `process/WebApiClient/` holds the failing workflow source. Grading: `skill_triggered` + `llm_judge` vs `RESOLUTION.md` (final response only). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/aa608d720c.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/aa608d720c.json deleted file mode 100644 index aa54e89d7b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/aa608d720c.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "automation1@test.com's workspace", - "Path": "automation1@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/bf6714348f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/bf6714348f.json deleted file mode 100644 index b59d56f49f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/bf6714348f.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error retrieving trace ID for job", - "Instructions": "Response returned an error code" -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/ce05d49d7f.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/ce05d49d7f.json deleted file mode 100644 index 6342e5ae14..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/ce05d49d7f.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "State": "Faulted", - "ProcessName": "FetchReport", - "ReleaseName": "FetchReport", - "StartTime": "2026-06-23T17:36:29.233Z", - "EndTime": "2026-06-23T17:36:32.110Z", - "CreationTime": "2026-06-23T17:36:28.890Z", - "Type": "Unattended", - "Source": "Manual", - "Priority": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/e2ec1940af.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/e2ec1940af.json deleted file mode 100644 index 39cfaf001f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/e2ec1940af.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:36:32.044Z", - "Message": "FetchReport execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-23T17:36:31.667Z", - "Message": "HTTP Request: The operation has timed out.\r\nin Wf_FetchReport.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Fetch Report\"\r\n at Wf_FetchReport \"Wf_FetchReport\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Net.WebException\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-23T17:36:29.479Z", - "Message": "FetchReport execution started" - } - ] -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/e80f19bddd.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/e80f19bddd.json deleted file mode 100644 index b7afeff40a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/e80f19bddd.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "StartTime": "2026-06-23T17:36:29.233Z", - "EndTime": "2026-06-23T17:36:32.110Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7076e54f-62c6-4bbe-acb5-48092340b366", - "Info": "The operation has timed out.\r\n\r\nin Wf_FetchReport.xaml\r\n at HttpClient \"HTTP Request\"\r\n at Sequence \"Fetch Report\"\r\n at Wf_FetchReport \"Wf_FetchReport\"\r\n\r\nSystem.Net.WebException: The operation has timed out.\r\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-23T17:36:28.890Z", - "ReleaseName": "FetchReport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 21569940, - "EntryPointPath": "Wf_FetchReport.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T17:36:32.110Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3bdb5a85b6ae4e1fb5a52488ca74de12", - "Id": 67889402 - } -} - diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/manifest.json deleted file mode 100644 index 0c3c3b2a4b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay of staging job 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12 (folder Shared). Real uip CLI evidence, scrubbed (host->MOCK-HOST, account->UIPATH\\AUTOMATION1).", - "rules": [ - { - "match": "or jobs logs 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12 --level Error", - "file": "75e53f025f.json" - }, - { - "match": "or jobs logs 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "file": "e2ec1940af.json" - }, - { - "match": "or jobs get 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "file": "e80f19bddd.json" - }, - { - "match": "or jobs history 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "file": "875d9f9c3a.json" - }, - { - "match": "or jobs traces 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "file": "9e4929c4d3.json" - }, - { - "match": "traces spans get --job-key 3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", - "file": "bf6714348f.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Faulted", - "file": "ce05d49d7f.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "ce05d49d7f.json" - }, - { - "match": "or jobs list", - "file": "ce05d49d7f.json" - }, - { - "match": "or folders list", - "file": "aa608d720c.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": { - "Result": "Success", - "Code": "Empty", - "Data": [] - }, - "exit_code": 0, - "_doc": "Empty result for any unmatched read-only uip query." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/uip-fixture.json new file mode 100644 index 0000000000..5a08cb432f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"FetchReport\",\r\n \"ReleaseName\": \"FetchReport\",\r\n \"StartTime\": \"2026-06-23T17:36:29.233Z\",\r\n \"EndTime\": \"2026-06-23T17:36:32.110Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:28.890Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:36:31.667Z\",\r\n \"Message\": \"HTTP Request: The operation has timed out.\\r\\nin Wf_FetchReport.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Fetch Report\\\"\\r\\n at Wf_FetchReport \\\"Wf_FetchReport\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error retrieving trace ID for job\",\r\n \"Instructions\": \"Response returned an error code\"\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"FetchReport\",\r\n \"ReleaseName\": \"FetchReport\",\r\n \"StartTime\": \"2026-06-23T17:36:29.233Z\",\r\n \"EndTime\": \"2026-06-23T17:36:32.110Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:28.890Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:36:32.044Z\",\r\n \"Message\": \"FetchReport execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T17:36:31.667Z\",\r\n \"Message\": \"HTTP Request: The operation has timed out.\\r\\nin Wf_FetchReport.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Fetch Report\\\"\\r\\n at Wf_FetchReport \\\"Wf_FetchReport\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Net.WebException\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-23T17:36:29.479Z\",\r\n \"Message\": \"FetchReport execution started\"\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12\",\r\n \"StartTime\": \"2026-06-23T17:36:29.233Z\",\r\n \"EndTime\": \"2026-06-23T17:36:32.110Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7076e54f-62c6-4bbe-acb5-48092340b366\",\r\n \"Info\": \"The operation has timed out.\\r\\n\\r\\nin Wf_FetchReport.xaml\\r\\n at HttpClient \\\"HTTP Request\\\"\\r\\n at Sequence \\\"Fetch Report\\\"\\r\\n at Wf_FetchReport \\\"Wf_FetchReport\\\"\\r\\n\\r\\nSystem.Net.WebException: The operation has timed out.\\r\\n at UiPath.Web.Activities.HttpClient.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-23T17:36:28.890Z\",\r\n \"ReleaseName\": \"FetchReport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 21569940,\r\n \"EntryPointPath\": \"Wf_FetchReport.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T17:36:32.110Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3bdb5a85b6ae4e1fb5a52488ca74de12\",\r\n \"Id\": 67889402\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-23T17:36:32.113Z\",\r\n \"JobId\": 67889402\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-23T17:36:29.240Z\",\r\n \"JobId\": 67889402\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-23T17:36:28.953Z\",\r\n \"JobId\": 67889402\r\n }\r\n ]\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3bdb5a85-b6ae-4e1f-b5a5-2488ca74de12\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"FetchReport\",\r\n \"ReleaseName\": \"FetchReport\",\r\n \"StartTime\": \"2026-06-23T17:36:29.233Z\",\r\n \"EndTime\": \"2026-06-23T17:36:32.110Z\",\r\n \"CreationTime\": \"2026-06-23T17:36:28.890Z\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"automation1@test.com's workspace\",\r\n \"Path\": \"automation1@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"Empty\",\n \"Data\": []\n}\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/task.yaml index b89b63a7cd..e717d7b062 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-http-timeout/task.yaml @@ -20,21 +20,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/README.md new file mode 100644 index 0000000000..2d5037b224 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/README.md @@ -0,0 +1,59 @@ +# Web Activities — MissingMethodException / Package-Studio Version Mismatch (design-time) + +This scenario reproduces a **design-time / validation** failure of a Web +Activities project: opening or validating the process, or clicking Configure +on the **HTTP Request wizard**, throws `System.MissingMethodException` — +"Method not found: 'Void UiPath.Web.Activities...'" — because `project.json` +pins `UiPath.WebAPI.Activities` at a version incompatible with the installed +Studio (and out of step with `UiPath.System.Activities`). The process has +**never run** — there are no Orchestrator jobs. + +## What this scenario uncovers + +**Root Cause:** `project.json` pins `UiPath.WebAPI.Activities [1.16.2]` against +`UiPath.System.Activities [21.10.5]` and `studioVersion 21.10.5.0` (Studio +2021.10). The WebAPI package build requires a newer Studio than the one +installed; its assembly loads but a member the wizard/activity was compiled +against is absent, so Studio throws `MissingMethodException` at design time. +The diagnosis comes from the **project source (the WebAPI package vs the +installed Studio)**, not job evidence. + +This maps to: +`references/activity-packages/web-activities/playbooks/package-version-mismatch.md`. + +The correct agent behavior is to reason from `project.json` (WebAPI version vs +Studio 2021.10 vs System.Activities) rather than chase Orchestrator jobs, +match the package-version-mismatch playbook, and recommend aligning the +packages to a Studio-compatible line via Manage Packages — explicitly NOT the +runtime "Could not load file or assembly" fix (clean +`%userprofile%\.nuget\packages` / republish), which does not apply because the +assembly loads and the failure reproduces in Studio at design time. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored Web Activities project (`HttpClient` / HTTP Request) pinning `UiPath.WebAPI.Activities [1.16.2]` vs `UiPath.System.Activities [21.10.5]` with `studioVersion 21.10.5.0` | + +> **Note on fixtures.** Like the `uia-dependency-version-conflict` and Word +> `replace-text-version-mismatch` scenarios, this has **no faulted job** by +> design — it is a design-time validation error, so `or jobs list` returns +> empty and the agent must diagnose from `project.json` + the Studio version +> in the prompt. It tests whether the agent distinguishes an in-project +> package/Studio version mismatch (design-time `MissingMethodException`) from +> the robot-side assembly-load failure ("Could not load file or assembly"). + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `package-version-mismatch.md` +- Agent identified the `UiPath.WebAPI.Activities` version as incompatible with + the installed Studio 2021.10 (and skewed vs `UiPath.System.Activities`) as + the design-time cause, reasoning from the project source, and recommended + aligning the packages to a Studio-compatible line via Manage Packages — + without confusing it with the runtime "cannot load assembly" + cache/republish fix or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/0a5723e777.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/0a5723e777.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/0a5723e777.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/2e37ebc77e.json deleted file mode 100644 index cd5bbf0630..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b", - "Name": "Integrations Dev", - "Path": "Integrations Dev", - "Description": "Development folder for integration automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f2a3b4c5-d6e7-4f8a-9b0c-1d2e3f4a5b6c", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/46e9213335.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/46e9213335.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/46e9213335.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/75949d659e.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/75949d659e.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/README.md deleted file mode 100644 index f0a8435546..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# Web Activities — MissingMethodException / Package-Studio Version Mismatch (design-time) - -This scenario reproduces a **design-time / validation** failure of a Web -Activities project: opening or validating the process, or clicking Configure -on the **HTTP Request wizard**, throws `System.MissingMethodException` — -"Method not found: 'Void UiPath.Web.Activities...'" — because `project.json` -pins `UiPath.WebAPI.Activities` at a version incompatible with the installed -Studio (and out of step with `UiPath.System.Activities`). The process has -**never run** — there are no Orchestrator jobs. - -## What this scenario uncovers - -**Root Cause:** `project.json` pins `UiPath.WebAPI.Activities [1.16.2]` against -`UiPath.System.Activities [21.10.5]` and `studioVersion 21.10.5.0` (Studio -2021.10). The WebAPI package build requires a newer Studio than the one -installed; its assembly loads but a member the wizard/activity was compiled -against is absent, so Studio throws `MissingMethodException` at design time. -The diagnosis comes from the **project source (the WebAPI package vs the -installed Studio)**, not job evidence. - -This maps to: -`references/activity-packages/web-activities/playbooks/package-version-mismatch.md`. - -The correct agent behavior is to reason from `project.json` (WebAPI version vs -Studio 2021.10 vs System.Activities) rather than chase Orchestrator jobs, -match the package-version-mismatch playbook, and recommend aligning the -packages to a Studio-compatible line via Manage Packages — explicitly NOT the -runtime "Could not load file or assembly" fix (clean -`%userprofile%\.nuget\packages` / republish), which does not apply because the -assembly loads and the failure reproduces in Studio at design time. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` dispatcher | shared from `../../_shared/mock_template/` | -| `process/` | hand-authored Web Activities project (`HttpClient` / HTTP Request) pinning `UiPath.WebAPI.Activities [1.16.2]` vs `UiPath.System.Activities [21.10.5]` with `studioVersion 21.10.5.0` | -| `data/m/r/*.json` | folders list + **empty** job lists (the process never ran) | -| `data/m/r/manifest.json` | dispatch table (folders, empty jobs, `docsai ask` passthrough, permissive `[]` fallback) | - -> **Note on fixtures.** Like the `uia-dependency-version-conflict` and Word -> `replace-text-version-mismatch` scenarios, this has **no faulted job** by -> design — it is a design-time validation error, so `or jobs list` returns -> empty and the agent must diagnose from `project.json` + the Studio version -> in the prompt. It tests whether the agent distinguishes an in-project -> package/Studio version mismatch (design-time `MissingMethodException`) from -> the robot-side assembly-load failure ("Could not load file or assembly"). - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `package-version-mismatch.md` -- Agent identified the `UiPath.WebAPI.Activities` version as incompatible with - the installed Studio 2021.10 (and skewed vs `UiPath.System.Activities`) as - the design-time cause, reasoning from the project source, and recommended - aligning the packages to a Studio-compatible line via Manage Packages — - without confusing it with the runtime "cannot load assembly" - cache/republish fix or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/a58aa4896d.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/a58aa4896d.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/a58aa4896d.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/c88524f965.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/c88524f965.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/c88524f965.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/manifest.json deleted file mode 100644 index f2bd092470..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatcher fixture set for this scenario; not investigation evidence. Provides a folder list and empty job lists (this process has no Orchestrator runs).", - "rules": [ - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key \"e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b\" --state Faulted --output json", - "file": "a58aa4896d.json" - }, - { - "match": "or jobs list --folder-key e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b --state Faulted --output json", - "file": "c88524f965.json" - }, - { - "match": "or jobs list --folder-key \"e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b\" --output json", - "file": "46e9213335.json" - }, - { - "match": "or jobs list --folder-key e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b --output json", - "file": "0a5723e777.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands (no jobs exist for this process)." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..cf21c9feb9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/data/uip-fixture.json @@ -0,0 +1,61 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e1f2a3b4-c5d6-4e7f-8a9b-0c1d2e3f4a5b\",\r\n \"Name\": \"Integrations Dev\",\r\n \"Path\": \"Integrations Dev\",\r\n \"Description\": \"Development folder for integration automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f2a3b4c5-d6e7-4f8a-9b0c-1d2e3f4a5b6c\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/task.yaml index 69229f5293..c59b6813d0 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/web-webapi-version-mismatch/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/59a50a86e9.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/59a50a86e9.json deleted file mode 100644 index 73767b024a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/59a50a86e9.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-23T14:20:02.140Z", - "EndTime": "2026-05-23T14:20:04.318Z", - "State": "Faulted", - "ReleaseName": "WordTemplateFill", - "Source": "Manual", - "Reference": "TemplateFillRun", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0033344-2222-3333-4444-555566667777", - "Info": "UiPath.Word.BusinessException: The bookmark 'LogoAnchor' was not found in the document.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/65acd77382.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/65acd77382.json deleted file mode 100644 index ff13299764..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/65acd77382.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0033344-2222-3333-4444-555566667777", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "Description": "Default shared automation folder.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6033001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/91ff04130c.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/91ff04130c.json deleted file mode 100644 index 542ddd55d7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/91ff04130c.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "State": "Pending", - "Timestamp": "2026-05-23T14:20:01.890Z" - }, - { - "Key": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "State": "Running", - "Timestamp": "2026-05-23T14:20:02.140Z" - }, - { - "Key": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "State": "Faulted", - "Timestamp": "2026-05-23T14:20:04.318Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/bd3788a51d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/bd3788a51d.json deleted file mode 100644 index 26f4d16d23..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/bd3788a51d.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-23T14:20:02.140Z", - "EndTime": "2026-05-23T14:20:04.318Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9cc33555-9999-aaaa-bbbb-ccccddddeeee", - "Info": "UiPath.Word.BusinessException: The bookmark 'LogoAnchor' was not found in the document.\r\n at UiPath.Word.Activities.WordAddImage.ResolveInsertLocation(Document doc)\r\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-23T14:20:01.890Z", - "ReleaseName": "WordTemplateFill", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680401, - "OrganizationUnitId": 6033001, - "FolderKey": "f0033344-2222-3333-4444-555566667777", - "Reference": "TemplateFillRun", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-23T14:20:04.318Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "BusinessException", - "ActivityName": "WordAddImage_1", - "ActivityDisplayName": "Add Picture", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "Id": 680077503 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/d05b032cfb.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/d05b032cfb.json deleted file mode 100644 index cc605109b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/d05b032cfb.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc335599-001", - "JobKey": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-23T14:20:02.180Z", - "Message": "Execution started.", - "ProcessName": "WordTemplateFill", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc335599-002", - "JobKey": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-23T14:20:02.402Z", - "Message": "[WordTemplateFill] Filling offer-letter template — inserting signature logo at the LogoAnchor bookmark.", - "ProcessName": "WordTemplateFill", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc335599-003", - "JobKey": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-23T14:20:03.640Z", - "Message": "Use Word File: offer-letter.docx — document opened (Path='C:\\Robot\\Templates\\offer-letter.docx').", - "ProcessName": "WordTemplateFill", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc335599-004", - "JobKey": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-23T14:20:04.110Z", - "Message": "Add Picture — resolving insertion point relative to bookmark 'LogoAnchor'.", - "ProcessName": "WordTemplateFill", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc335599-005", - "JobKey": "cc335555-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Error", - "Timestamp": "2026-05-23T14:20:04.290Z", - "Message": "Add Picture: UiPath.Word.BusinessException: The bookmark 'LogoAnchor' was not found in the document.", - "ProcessName": "WordTemplateFill", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/manifest.json deleted file mode 100644 index 608d1790e0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Add Picture bookmark-missing scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "65acd77382.json" - }, - { - "match": "or jobs list", - "file": "59a50a86e9.json" - }, - { - "match": "or jobs logs", - "file": "d05b032cfb.json" - }, - { - "match": "or jobs history", - "file": "91ff04130c.json" - }, - { - "match": "or jobs get", - "file": "bd3788a51d.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/uip-fixture.json new file mode 100644 index 0000000000..248c5e76aa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0033344-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"Description\": \"Default shared automation folder.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6033001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-23T14:20:02.140Z\",\r\n \"EndTime\": \"2026-05-23T14:20:04.318Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"WordTemplateFill\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"TemplateFillRun\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0033344-2222-3333-4444-555566667777\",\r\n \"Info\": \"UiPath.Word.BusinessException: The bookmark 'LogoAnchor' was not found in the document.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc335599-001\",\r\n \"JobKey\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-23T14:20:02.180Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"WordTemplateFill\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc335599-002\",\r\n \"JobKey\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-23T14:20:02.402Z\",\r\n \"Message\": \"[WordTemplateFill] Filling offer-letter template — inserting signature logo at the LogoAnchor bookmark.\",\r\n \"ProcessName\": \"WordTemplateFill\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc335599-003\",\r\n \"JobKey\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T14:20:03.640Z\",\r\n \"Message\": \"Use Word File: offer-letter.docx — document opened (Path='C:\\\\Robot\\\\Templates\\\\offer-letter.docx').\",\r\n \"ProcessName\": \"WordTemplateFill\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc335599-004\",\r\n \"JobKey\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-23T14:20:04.110Z\",\r\n \"Message\": \"Add Picture — resolving insertion point relative to bookmark 'LogoAnchor'.\",\r\n \"ProcessName\": \"WordTemplateFill\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc335599-005\",\r\n \"JobKey\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-23T14:20:04.290Z\",\r\n \"Message\": \"Add Picture: UiPath.Word.BusinessException: The bookmark 'LogoAnchor' was not found in the document.\",\r\n \"ProcessName\": \"WordTemplateFill\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-23T14:20:01.890Z\"\r\n },\r\n {\r\n \"Key\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-23T14:20:02.140Z\"\r\n },\r\n {\r\n \"Key\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-23T14:20:04.318Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-23T14:20:02.140Z\",\r\n \"EndTime\": \"2026-05-23T14:20:04.318Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9cc33555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Info\": \"UiPath.Word.BusinessException: The bookmark 'LogoAnchor' was not found in the document.\\r\\n at UiPath.Word.Activities.WordAddImage.ResolveInsertLocation(Document doc)\\r\\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-23T14:20:01.890Z\",\r\n \"ReleaseName\": \"WordTemplateFill\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680401,\r\n \"OrganizationUnitId\": 6033001,\r\n \"FolderKey\": \"f0033344-2222-3333-4444-555566667777\",\r\n \"Reference\": \"TemplateFillRun\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-23T14:20:04.318Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"BusinessException\",\r\n \"ActivityName\": \"WordAddImage_1\",\r\n \"ActivityDisplayName\": \"Add Picture\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc335555-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Id\": 680077503\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/task.yaml index 605f1c095d..38c8329070 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-bookmark-missing/task.yaml @@ -30,21 +30,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/1cf6f122df.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/1cf6f122df.json deleted file mode 100644 index 07abb2a1c5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/1cf6f122df.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "State": "Pending", - "Timestamp": "2026-05-21T11:05:01.890Z" - }, - { - "Key": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "State": "Running", - "Timestamp": "2026-05-21T11:05:02.140Z" - }, - { - "Key": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "State": "Faulted", - "Timestamp": "2026-05-21T11:05:05.118Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/46ecdd96cb.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/46ecdd96cb.json deleted file mode 100644 index d02e4b2057..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/46ecdd96cb.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-bb222299-001", - "JobKey": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-21T11:05:02.180Z", - "Message": "Execution started.", - "ProcessName": "WordContractStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222299-002", - "JobKey": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-21T11:05:02.402Z", - "Message": "[WordContractStamp] Stamping company logo into the contract draft.", - "ProcessName": "WordContractStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222299-003", - "JobKey": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-21T11:05:04.640Z", - "Message": "Use Word File: contract-draft.docx — launching Word via Office Interop (Visible=False) for document 'C:\\Robot\\Contracts\\contract-draft.docx'.", - "ProcessName": "WordContractStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-bb222299-004", - "JobKey": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Error", - "Timestamp": "2026-05-21T11:05:05.090Z", - "Message": "Add Picture: System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Word.ApplicationClass' to interface type 'Microsoft.Office.Interop.Word._Application'. ... (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).", - "ProcessName": "WordContractStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/bd06ef581f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/bd06ef581f.json deleted file mode 100644 index c97b2d431a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/bd06ef581f.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-21T11:05:02.140Z", - "EndTime": "2026-05-21T11:05:05.118Z", - "State": "Faulted", - "ReleaseName": "WordContractStamp", - "Source": "Manual", - "Reference": "ContractStampRun", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022244-2222-3333-4444-555566667777", - "Info": "System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Word.ApplicationClass' to interface type 'Microsoft.Office.Interop.Word._Application'. ... (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/ce587f2a7b.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/ce587f2a7b.json deleted file mode 100644 index 34a62544b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/ce587f2a7b.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022244-2222-3333-4444-555566667777", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "Description": "Default shared automation folder.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6022001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/f7f63c084a.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/f7f63c084a.json deleted file mode 100644 index fffcfd9863..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/f7f63c084a.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-21T11:05:02.140Z", - "EndTime": "2026-05-21T11:05:05.118Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9bb22222-9999-aaaa-bbbb-ccccddddeeee", - "Info": "System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Word.ApplicationClass' to interface type 'Microsoft.Office.Interop.Word._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00020970-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\r\n at System.RuntimeType.ForwardCallToInvokeMember(String memberName, BindingFlags flags, Object target, Int32[] aWrapperTypes, MessageData& msgData)\r\n at UiPath.Word.Activities.WordApplicationScopeRuntime.EnsureWordApplication()\r\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-21T11:05:01.890Z", - "ReleaseName": "WordContractStamp", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680201, - "OrganizationUnitId": 6022001, - "FolderKey": "f0022244-2222-3333-4444-555566667777", - "Reference": "ContractStampRun", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-21T11:05:05.118Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidCastException", - "ActivityName": "WordAddImage_1", - "ActivityDisplayName": "Add Picture", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "bb222222-9999-aaaa-bbbb-ccccddddeeee", - "Id": 680077201 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/manifest.json deleted file mode 100644 index 9c6b7f23a9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Add Picture com-interop scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "ce587f2a7b.json" - }, - { - "match": "or jobs list", - "file": "bd06ef581f.json" - }, - { - "match": "or jobs logs", - "file": "46ecdd96cb.json" - }, - { - "match": "or jobs history", - "file": "1cf6f122df.json" - }, - { - "match": "or jobs get", - "file": "f7f63c084a.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/uip-fixture.json new file mode 100644 index 0000000000..6dd1bde3bf --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022244-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"Description\": \"Default shared automation folder.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6022001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-21T11:05:02.140Z\",\r\n \"EndTime\": \"2026-05-21T11:05:05.118Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"WordContractStamp\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"ContractStampRun\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022244-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Word.ApplicationClass' to interface type 'Microsoft.Office.Interop.Word._Application'. ... (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-bb222299-001\",\r\n \"JobKey\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-21T11:05:02.180Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"WordContractStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222299-002\",\r\n \"JobKey\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-21T11:05:02.402Z\",\r\n \"Message\": \"[WordContractStamp] Stamping company logo into the contract draft.\",\r\n \"ProcessName\": \"WordContractStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222299-003\",\r\n \"JobKey\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-21T11:05:04.640Z\",\r\n \"Message\": \"Use Word File: contract-draft.docx — launching Word via Office Interop (Visible=False) for document 'C:\\\\Robot\\\\Contracts\\\\contract-draft.docx'.\",\r\n \"ProcessName\": \"WordContractStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-bb222299-004\",\r\n \"JobKey\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-21T11:05:05.090Z\",\r\n \"Message\": \"Add Picture: System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Word.ApplicationClass' to interface type 'Microsoft.Office.Interop.Word._Application'. ... (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\",\r\n \"ProcessName\": \"WordContractStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-21T11:05:01.890Z\"\r\n },\r\n {\r\n \"Key\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-21T11:05:02.140Z\"\r\n },\r\n {\r\n \"Key\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-21T11:05:05.118Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-21T11:05:02.140Z\",\r\n \"EndTime\": \"2026-05-21T11:05:05.118Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9bb22222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Info\": \"System.InvalidCastException: Unable to cast COM object of type 'Microsoft.Office.Interop.Word.ApplicationClass' to interface type 'Microsoft.Office.Interop.Word._Application'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{00020970-0000-0000-C000-000000000046}' failed due to the following error: Library not registered. (Exception from HRESULT: 0x8002801D (TYPE_E_LIBNOTREGISTERED)).\\r\\n at System.RuntimeType.ForwardCallToInvokeMember(String memberName, BindingFlags flags, Object target, Int32[] aWrapperTypes, MessageData& msgData)\\r\\n at UiPath.Word.Activities.WordApplicationScopeRuntime.EnsureWordApplication()\\r\\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-21T11:05:01.890Z\",\r\n \"ReleaseName\": \"WordContractStamp\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680201,\r\n \"OrganizationUnitId\": 6022001,\r\n \"FolderKey\": \"f0022244-2222-3333-4444-555566667777\",\r\n \"Reference\": \"ContractStampRun\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-21T11:05:05.118Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidCastException\",\r\n \"ActivityName\": \"WordAddImage_1\",\r\n \"ActivityDisplayName\": \"Add Picture\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"bb222222-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Id\": 680077201\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/task.yaml index 45d4b59417..ba417e0118 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-com-interop/task.yaml @@ -31,21 +31,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/06f10eeac5.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/06f10eeac5.json deleted file mode 100644 index c7fe72d4bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/06f10eeac5.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0044444-2222-3333-4444-555566667777", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "Description": "Default shared automation folder.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6040001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/8c8e3fd166.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/8c8e3fd166.json deleted file mode 100644 index f4985d7a53..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/8c8e3fd166.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "State": "Pending", - "Timestamp": "2026-05-22T09:15:01.890Z" - }, - { - "Key": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "State": "Running", - "Timestamp": "2026-05-22T09:15:02.140Z" - }, - { - "Key": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "State": "Faulted", - "Timestamp": "2026-05-22T09:15:04.302Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/9bb0c38641.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/9bb0c38641.json deleted file mode 100644 index e3939ae29d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/9bb0c38641.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-22T09:15:02.140Z", - "EndTime": "2026-05-22T09:15:04.302Z", - "State": "Faulted", - "ReleaseName": "WordReportImage", - "Source": "Manual", - "Reference": "MonthlyReportRun", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0044444-2222-3333-4444-555566667777", - "Info": "System.IO.FileNotFoundException: Could not find file 'C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\WordReportImage\\1.0.0\\UiPath.Core.Image'.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/a5e38effc8.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/a5e38effc8.json deleted file mode 100644 index 0e0e587dc5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/a5e38effc8.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-22T09:15:02.140Z", - "EndTime": "2026-05-22T09:15:04.302Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9dd44444-9999-aaaa-bbbb-ccccddddeeee", - "Info": "System.IO.FileNotFoundException: Could not find file 'C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\WordReportImage\\1.0.0\\UiPath.Core.Image'.\r\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\r\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, ...)\r\n at UiPath.Word.Activities.WordAddImage.AddImageFromPath(String imagePath)\r\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-22T09:15:01.890Z", - "ReleaseName": "WordReportImage", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680401, - "OrganizationUnitId": 6040001, - "FolderKey": "f0044444-2222-3333-4444-555566667777", - "Reference": "MonthlyReportRun", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-22T09:15:04.302Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "FileNotFoundException", - "ActivityName": "WordAddImage_1", - "ActivityDisplayName": "Add Picture", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "Id": 680077403 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/ed194221a4.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/ed194221a4.json deleted file mode 100644 index 14115c26e2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/ed194221a4.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-dd444499-001", - "JobKey": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-22T09:15:02.180Z", - "Message": "Execution started.", - "ProcessName": "WordReportImage", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444499-002", - "JobKey": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-22T09:15:02.402Z", - "Message": "[WordReportImage] Building monthly report — capturing chart screenshot and inserting into the document.", - "ProcessName": "WordReportImage", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444499-003", - "JobKey": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-22T09:15:03.640Z", - "Message": "Use Word File: monthly-report.docx — document opened (Path='C:\\Robot\\Reports\\monthly-report.docx').", - "ProcessName": "WordReportImage", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444499-004", - "JobKey": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-22T09:15:04.110Z", - "Message": "Add Picture — inserting image from path 'UiPath.Core.Image' (ImagePath expression resolved to this string).", - "ProcessName": "WordReportImage", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-dd444499-005", - "JobKey": "dd444444-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Error", - "Timestamp": "2026-05-22T09:15:04.290Z", - "Message": "Add Picture: System.IO.FileNotFoundException: Could not find file 'C:\\Users\\automation1\\AppData\\Local\\UiPath\\Packages\\WordReportImage\\1.0.0\\UiPath.Core.Image'.", - "ProcessName": "WordReportImage", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/manifest.json deleted file mode 100644 index 8e7b7e1391..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Add Picture image-variable scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "06f10eeac5.json" - }, - { - "match": "or jobs list", - "file": "9bb0c38641.json" - }, - { - "match": "or jobs logs", - "file": "ed194221a4.json" - }, - { - "match": "or jobs history", - "file": "8c8e3fd166.json" - }, - { - "match": "or jobs get", - "file": "a5e38effc8.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/uip-fixture.json new file mode 100644 index 0000000000..0ba88049ea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0044444-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"Description\": \"Default shared automation folder.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6040001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-22T09:15:02.140Z\",\r\n \"EndTime\": \"2026-05-22T09:15:04.302Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"WordReportImage\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"MonthlyReportRun\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0044444-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.IO.FileNotFoundException: Could not find file 'C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\WordReportImage\\\\1.0.0\\\\UiPath.Core.Image'.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-dd444499-001\",\r\n \"JobKey\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-22T09:15:02.180Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"WordReportImage\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444499-002\",\r\n \"JobKey\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-22T09:15:02.402Z\",\r\n \"Message\": \"[WordReportImage] Building monthly report — capturing chart screenshot and inserting into the document.\",\r\n \"ProcessName\": \"WordReportImage\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444499-003\",\r\n \"JobKey\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T09:15:03.640Z\",\r\n \"Message\": \"Use Word File: monthly-report.docx — document opened (Path='C:\\\\Robot\\\\Reports\\\\monthly-report.docx').\",\r\n \"ProcessName\": \"WordReportImage\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444499-004\",\r\n \"JobKey\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T09:15:04.110Z\",\r\n \"Message\": \"Add Picture — inserting image from path 'UiPath.Core.Image' (ImagePath expression resolved to this string).\",\r\n \"ProcessName\": \"WordReportImage\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-dd444499-005\",\r\n \"JobKey\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-22T09:15:04.290Z\",\r\n \"Message\": \"Add Picture: System.IO.FileNotFoundException: Could not find file 'C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\WordReportImage\\\\1.0.0\\\\UiPath.Core.Image'.\",\r\n \"ProcessName\": \"WordReportImage\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-22T09:15:01.890Z\"\r\n },\r\n {\r\n \"Key\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-22T09:15:02.140Z\"\r\n },\r\n {\r\n \"Key\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-22T09:15:04.302Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-22T09:15:02.140Z\",\r\n \"EndTime\": \"2026-05-22T09:15:04.302Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9dd44444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Info\": \"System.IO.FileNotFoundException: Could not find file 'C:\\\\Users\\\\automation1\\\\AppData\\\\Local\\\\UiPath\\\\Packages\\\\WordReportImage\\\\1.0.0\\\\UiPath.Core.Image'.\\r\\n at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, ...)\\r\\n at UiPath.Word.Activities.WordAddImage.AddImageFromPath(String imagePath)\\r\\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-22T09:15:01.890Z\",\r\n \"ReleaseName\": \"WordReportImage\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680401,\r\n \"OrganizationUnitId\": 6040001,\r\n \"FolderKey\": \"f0044444-2222-3333-4444-555566667777\",\r\n \"Reference\": \"MonthlyReportRun\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-22T09:15:04.302Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"FileNotFoundException\",\r\n \"ActivityName\": \"WordAddImage_1\",\r\n \"ActivityDisplayName\": \"Add Picture\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"dd444444-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Id\": 680077403\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/task.yaml index 54a6d2be9b..13f31cd828 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-image-variable/task.yaml @@ -28,21 +28,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/331559ee2e.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/331559ee2e.json deleted file mode 100644 index 9833d7a5b2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/331559ee2e.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-19T07:30:02.115Z", - "EndTime": "2026-05-19T07:30:03.061Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9aa11111-9999-aaaa-bbbb-ccccddddeeee", - "Info": "System.InvalidOperationException: The 'Add Picture' activity must be used inside a 'Use Word File' or 'Word Application Scope' activity. No Word document is open in the current scope.\r\n at UiPath.Word.Activities.WordAddImage.ResolveDocumentContext()\r\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\r\n", - "CreationTime": "2026-05-19T07:30:01.870Z", - "ReleaseName": "WordLogoStamp", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5610401, - "OrganizationUnitId": 6011001, - "FolderKey": "f0011144-2222-3333-4444-555566667777", - "Reference": "LogoStampRun", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T07:30:03.061Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidOperationException", - "ActivityName": "WordAddImage_1", - "ActivityDisplayName": "Add Picture", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Id": 680011403 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/50192be51f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/50192be51f.json deleted file mode 100644 index d15c70bcc6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/50192be51f.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-aa111199-001", - "JobKey": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-19T07:30:02.180Z", - "Message": "Execution started.", - "ProcessName": "WordLogoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111199-002", - "JobKey": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-19T07:30:02.402Z", - "Message": "[WordLogoStamp] Stamping the company logo into the monthly report.", - "ProcessName": "WordLogoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111199-003", - "JobKey": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-19T07:30:02.640Z", - "Message": "Use Word File: monthly-report.docx — document opened (Path='C:\\Robot\\Reports\\monthly-report.docx').", - "ProcessName": "WordLogoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111199-004", - "JobKey": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Info", - "Timestamp": "2026-05-19T07:30:02.710Z", - "Message": "[WordLogoStamp] Report document opened — ready to stamp logo.", - "ProcessName": "WordLogoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111199-005", - "JobKey": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Trace", - "Timestamp": "2026-05-19T07:30:02.880Z", - "Message": "Use Word File: monthly-report.docx — scope exited; document context closed.", - "ProcessName": "WordLogoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-aa111199-006", - "JobKey": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "Level": "Error", - "Timestamp": "2026-05-19T07:30:03.050Z", - "Message": "Add Picture: System.InvalidOperationException: The 'Add Picture' activity must be used inside a 'Use Word File' or 'Word Application Scope' activity. No Word document is open in the current scope.", - "ProcessName": "WordLogoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/756f2d29c6.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/756f2d29c6.json deleted file mode 100644 index e32cae8cf3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/756f2d29c6.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "State": "Pending", - "Timestamp": "2026-05-19T07:30:01.870Z" - }, - { - "Key": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "State": "Running", - "Timestamp": "2026-05-19T07:30:02.115Z" - }, - { - "Key": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "State": "Faulted", - "Timestamp": "2026-05-19T07:30:03.061Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/9409b55e6d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/9409b55e6d.json deleted file mode 100644 index c9f1765cb9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/9409b55e6d.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "aa111111-9999-aaaa-bbbb-ccccddddeeee", - "StartTime": "2026-05-19T07:30:02.115Z", - "EndTime": "2026-05-19T07:30:03.061Z", - "State": "Faulted", - "ReleaseName": "WordLogoStamp", - "Source": "Manual", - "Reference": "LogoStampRun", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0011144-2222-3333-4444-555566667777", - "Info": "System.InvalidOperationException: The 'Add Picture' activity must be used inside a 'Use Word File' or 'Word Application Scope' activity. No Word document is open in the current scope.", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/9b78c7addc.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/9b78c7addc.json deleted file mode 100644 index 09214f6318..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/9b78c7addc.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0011144-2222-3333-4444-555566667777", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "Description": "Default shared automation folder.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6011001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/manifest.json deleted file mode 100644 index 66f8e6af32..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Add Picture missing-scope scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "9b78c7addc.json" - }, - { - "match": "or jobs list", - "file": "9409b55e6d.json" - }, - { - "match": "or jobs logs", - "file": "50192be51f.json" - }, - { - "match": "or jobs history", - "file": "756f2d29c6.json" - }, - { - "match": "or jobs get", - "file": "331559ee2e.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/uip-fixture.json new file mode 100644 index 0000000000..1ccb16c0ad --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0011144-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"Description\": \"Default shared automation folder.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6011001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-19T07:30:02.115Z\",\r\n \"EndTime\": \"2026-05-19T07:30:03.061Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"WordLogoStamp\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"LogoStampRun\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0011144-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.InvalidOperationException: The 'Add Picture' activity must be used inside a 'Use Word File' or 'Word Application Scope' activity. No Word document is open in the current scope.\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-aa111199-001\",\r\n \"JobKey\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T07:30:02.180Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"WordLogoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111199-002\",\r\n \"JobKey\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T07:30:02.402Z\",\r\n \"Message\": \"[WordLogoStamp] Stamping the company logo into the monthly report.\",\r\n \"ProcessName\": \"WordLogoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111199-003\",\r\n \"JobKey\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T07:30:02.640Z\",\r\n \"Message\": \"Use Word File: monthly-report.docx — document opened (Path='C:\\\\Robot\\\\Reports\\\\monthly-report.docx').\",\r\n \"ProcessName\": \"WordLogoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111199-004\",\r\n \"JobKey\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-19T07:30:02.710Z\",\r\n \"Message\": \"[WordLogoStamp] Report document opened — ready to stamp logo.\",\r\n \"ProcessName\": \"WordLogoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111199-005\",\r\n \"JobKey\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-19T07:30:02.880Z\",\r\n \"Message\": \"Use Word File: monthly-report.docx — scope exited; document context closed.\",\r\n \"ProcessName\": \"WordLogoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-aa111199-006\",\r\n \"JobKey\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-19T07:30:03.050Z\",\r\n \"Message\": \"Add Picture: System.InvalidOperationException: The 'Add Picture' activity must be used inside a 'Use Word File' or 'Word Application Scope' activity. No Word document is open in the current scope.\",\r\n \"ProcessName\": \"WordLogoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-19T07:30:01.870Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-19T07:30:02.115Z\"\r\n },\r\n {\r\n \"Key\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-19T07:30:03.061Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"StartTime\": \"2026-05-19T07:30:02.115Z\",\r\n \"EndTime\": \"2026-05-19T07:30:03.061Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9aa11111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Info\": \"System.InvalidOperationException: The 'Add Picture' activity must be used inside a 'Use Word File' or 'Word Application Scope' activity. No Word document is open in the current scope.\\r\\n at UiPath.Word.Activities.WordAddImage.ResolveDocumentContext()\\r\\n at UiPath.Word.Activities.WordAddImage.Execute(NativeActivityContext context)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T07:30:01.870Z\",\r\n \"ReleaseName\": \"WordLogoStamp\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5610401,\r\n \"OrganizationUnitId\": 6011001,\r\n \"FolderKey\": \"f0011144-2222-3333-4444-555566667777\",\r\n \"Reference\": \"LogoStampRun\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T07:30:03.061Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidOperationException\",\r\n \"ActivityName\": \"WordAddImage_1\",\r\n \"ActivityDisplayName\": \"Add Picture\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"aa111111-9999-aaaa-bbbb-ccccddddeeee\",\r\n \"Id\": 680011403\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/task.yaml index d27a3db19b..75edee47d4 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-missing-scope/task.yaml @@ -27,21 +27,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/6618893ff1.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/6618893ff1.json deleted file mode 100644 index e0028c763e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/6618893ff1.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-05-22T09:14:02.300Z", - "EndTime": "2026-05-22T09:14:08.742Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9cc33333-3333-bbbb-cccc-ddddeeeeffff", - "Info": "System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)).\r\n at UiPath.Word.Activities.WordInteropActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n", - "CreationTime": "2026-05-22T09:14:01.980Z", - "ReleaseName": "ResumePhotoStamp", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5680301, - "OrganizationUnitId": 6022001, - "FolderKey": "f0022244-2222-3333-4444-555566667777", - "Reference": "ResumePhotoRun", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-22T09:14:08.742Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidCastException", - "ActivityName": "WordAddImage_1", - "ActivityDisplayName": "Add Picture", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Id": 680077301 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/88a092496d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/88a092496d.json deleted file mode 100644 index 3c7c637524..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/88a092496d.json +++ /dev/null @@ -1,66 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Key": "log-cc333399-001", - "JobKey": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-05-22T09:14:02.360Z", - "Message": "Execution started.", - "ProcessName": "ResumePhotoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-002", - "JobKey": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Level": "Info", - "Timestamp": "2026-05-22T09:14:02.640Z", - "Message": "[ResumePhotoStamp] Inserting the candidate photo into the resume draft.", - "ProcessName": "ResumePhotoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-003", - "JobKey": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Level": "Trace", - "Timestamp": "2026-05-22T09:14:04.210Z", - "Message": "Use Word File: resume-draft.docx — launching Word via Office Interop (Visible=False) for document 'C:\\Robot\\Resumes\\resume-draft.docx'. Document opened successfully.", - "ProcessName": "ResumePhotoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-004", - "JobKey": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Level": "Trace", - "Timestamp": "2026-05-22T09:14:07.880Z", - "Message": "Add Picture: inserting image 'C:\\Robot\\Assets\\candidate-photo.jpg' relative to Document (Position=End).", - "ProcessName": "ResumePhotoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-005", - "JobKey": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Level": "Warn", - "Timestamp": "2026-05-22T09:14:08.520Z", - "Message": "Word Application host process (WINWORD.EXE, PID 7864) exited unexpectedly while 'Add Picture' was inserting the image. The Word COM server is no longer available.", - "ProcessName": "ResumePhotoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - }, - { - "Key": "log-cc333399-006", - "JobKey": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "Level": "Error", - "Timestamp": "2026-05-22T09:14:08.700Z", - "Message": "Add Picture: System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. ... The application called an interface that was marshalled for a different thread. (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)). Thrown from UiPath.Word.Activities.WordInteropActivity.EndExecute.", - "ProcessName": "ResumePhotoStamp", - "MachineName": "MOCK-HOST", - "RobotName": "AUTOMATION1" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/cd2a0f832f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/cd2a0f832f.json deleted file mode 100644 index 782a7b392f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/cd2a0f832f.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "Key": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "State": "Pending", - "Timestamp": "2026-05-22T09:14:01.980Z" - }, - { - "Key": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "State": "Running", - "Timestamp": "2026-05-22T09:14:02.300Z" - }, - { - "Key": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "State": "Faulted", - "Timestamp": "2026-05-22T09:14:08.742Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/ce587f2a7b.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/ce587f2a7b.json deleted file mode 100644 index 34a62544b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/ce587f2a7b.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "FoldersListCurrentUser", - "Data": [ - { - "Key": "f0022244-2222-3333-4444-555566667777", - "DisplayName": "Shared", - "FullyQualifiedName": "Shared", - "Description": "Default shared automation folder.", - "ProvisionType": "Manual", - "PermissionModel": "FineGrained", - "FeedType": "Processes", - "Id": 6022001, - "ParentId": null, - "ParentKey": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/f160025a12.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/f160025a12.json deleted file mode 100644 index eb2515e9be..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/f160025a12.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "cc333333-3333-bbbb-cccc-ddddeeeeffff", - "StartTime": "2026-05-22T09:14:02.300Z", - "EndTime": "2026-05-22T09:14:08.742Z", - "State": "Faulted", - "ReleaseName": "ResumePhotoStamp", - "Source": "Manual", - "Reference": "ResumePhotoRun", - "HostMachineName": "MOCK-HOST", - "FolderKey": "f0022244-2222-3333-4444-555566667777", - "Info": "System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. ... (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)).", - "ErrorCode": "Activity" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/manifest.json deleted file mode 100644 index b891089bce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Add Picture word-crash scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - }, - { - "match": "or folders list", - "file": "ce587f2a7b.json" - }, - { - "match": "or jobs list", - "file": "f160025a12.json" - }, - { - "match": "or jobs logs", - "file": "88a092496d.json" - }, - { - "match": "or jobs history", - "file": "cd2a0f832f.json" - }, - { - "match": "or jobs get", - "file": "6618893ff1.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/uip-fixture.json new file mode 100644 index 0000000000..6f939d79c7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/data/uip-fixture.json @@ -0,0 +1,59 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FoldersListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0022244-2222-3333-4444-555566667777\",\r\n \"DisplayName\": \"Shared\",\r\n \"FullyQualifiedName\": \"Shared\",\r\n \"Description\": \"Default shared automation folder.\",\r\n \"ProvisionType\": \"Manual\",\r\n \"PermissionModel\": \"FineGrained\",\r\n \"FeedType\": \"Processes\",\r\n \"Id\": 6022001,\r\n \"ParentId\": null,\r\n \"ParentKey\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-05-22T09:14:02.300Z\",\r\n \"EndTime\": \"2026-05-22T09:14:08.742Z\",\r\n \"State\": \"Faulted\",\r\n \"ReleaseName\": \"ResumePhotoStamp\",\r\n \"Source\": \"Manual\",\r\n \"Reference\": \"ResumePhotoRun\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"f0022244-2222-3333-4444-555566667777\",\r\n \"Info\": \"System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. ... (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)).\",\r\n \"ErrorCode\": \"Activity\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"log-cc333399-001\",\r\n \"JobKey\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-22T09:14:02.360Z\",\r\n \"Message\": \"Execution started.\",\r\n \"ProcessName\": \"ResumePhotoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-002\",\r\n \"JobKey\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Info\",\r\n \"Timestamp\": \"2026-05-22T09:14:02.640Z\",\r\n \"Message\": \"[ResumePhotoStamp] Inserting the candidate photo into the resume draft.\",\r\n \"ProcessName\": \"ResumePhotoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-003\",\r\n \"JobKey\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T09:14:04.210Z\",\r\n \"Message\": \"Use Word File: resume-draft.docx — launching Word via Office Interop (Visible=False) for document 'C:\\\\Robot\\\\Resumes\\\\resume-draft.docx'. Document opened successfully.\",\r\n \"ProcessName\": \"ResumePhotoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-004\",\r\n \"JobKey\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Trace\",\r\n \"Timestamp\": \"2026-05-22T09:14:07.880Z\",\r\n \"Message\": \"Add Picture: inserting image 'C:\\\\Robot\\\\Assets\\\\candidate-photo.jpg' relative to Document (Position=End).\",\r\n \"ProcessName\": \"ResumePhotoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-005\",\r\n \"JobKey\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Warn\",\r\n \"Timestamp\": \"2026-05-22T09:14:08.520Z\",\r\n \"Message\": \"Word Application host process (WINWORD.EXE, PID 7864) exited unexpectedly while 'Add Picture' was inserting the image. The Word COM server is no longer available.\",\r\n \"ProcessName\": \"ResumePhotoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n },\r\n {\r\n \"Key\": \"log-cc333399-006\",\r\n \"JobKey\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Level\": \"Error\",\r\n \"Timestamp\": \"2026-05-22T09:14:08.700Z\",\r\n \"Message\": \"Add Picture: System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. ... The application called an interface that was marshalled for a different thread. (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)). Thrown from UiPath.Word.Activities.WordInteropActivity.EndExecute.\",\r\n \"ProcessName\": \"ResumePhotoStamp\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"RobotName\": \"AUTOMATION1\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-22T09:14:01.980Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-22T09:14:02.300Z\"\r\n },\r\n {\r\n \"Key\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-22T09:14:08.742Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"StartTime\": \"2026-05-22T09:14:02.300Z\",\r\n \"EndTime\": \"2026-05-22T09:14:08.742Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9cc33333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Info\": \"System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (Exception from HRESULT: 0x8001010E (RPC_E_WRONG_THREAD)).\\r\\n at UiPath.Word.Activities.WordInteropActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n\",\r\n \"CreationTime\": \"2026-05-22T09:14:01.980Z\",\r\n \"ReleaseName\": \"ResumePhotoStamp\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5680301,\r\n \"OrganizationUnitId\": 6022001,\r\n \"FolderKey\": \"f0022244-2222-3333-4444-555566667777\",\r\n \"Reference\": \"ResumePhotoRun\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-22T09:14:08.742Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidCastException\",\r\n \"ActivityName\": \"WordAddImage_1\",\r\n \"ActivityDisplayName\": \"Add Picture\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"cc333333-3333-bbbb-cccc-ddddeeeeffff\",\r\n \"Id\": 680077301\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/task.yaml index e0f75456fd..3bf9c03b4f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-addpicture-word-crash/task.yaml @@ -35,21 +35,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/README.md new file mode 100644 index 0000000000..3520763733 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/README.md @@ -0,0 +1,46 @@ +# Append Text Failure - Activity Outside Its Container + +This scenario reproduces an `Append Text` failure where the App-Integration +`Append Text` activity is dropped **outside** a `Word Application Scope` and +faults with `Activity is valid only inside WordApplicationScope` — it has no +file input of its own. + +## What this scenario uncovers + +**Root Cause:** The App-Integration `Append Text` (`WordAppendText`) sits +loose in the sequence with no container. It appends to the document held open +by a surrounding scope; outside one it is invalid, so the job faults +immediately. The file/document/install are not implicated. + +This maps to: +`references/activity-packages/word-activities/playbooks/append-text-missing-container.md` + +The correct agent behavior is to tie the fault to the missing container / +wrong Append Text surface and recommend either nesting it in a +`Word Application Scope` / `Use Word File` or switching to the standalone +`Word Document` `Append Text` (which takes a file path) — not blaming the +file path, document, or install. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a `Word Application Scope`-less `Append Text` loose in the sequence | + +> **Note on fixtures.** The faulted job's error names the `Append Text` +> activity and the `WordApplicationScope` requirement; the `.xaml` shows the +> activity loose in the `Main Sequence`. Twin of `read-text-missing-container` +> for a different activity. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `append-text-missing-container.md` +- Agent identified the App-Integration Append Text being outside a + `Word Application Scope` as the cause and recommended nesting it in a scope + OR switching to the standalone `Word Document` `Append Text` — without + blaming the file/document/install or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/0f34c8f658.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/0f34c8f658.json deleted file mode 100644 index 8f0d4bfc2e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/0f34c8f658.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8", - "StartTime": "2026-06-17T10:20:01.330Z", - "EndTime": "2026-06-17T10:20:02.210Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c2d3e4f5-a6b7-4315-9283-94a5b6c7d8e9", - "Info": "Validation Error: Activity 'Append Text' is valid only inside WordApplicationScope.\r\n\r\nin Main.xaml\r\n at WordAppendText \"Append Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Word.Activities.WordException: Activity is valid only inside WordApplicationScope.", - "CreationTime": "2026-06-17T10:20:01.100Z", - "ReleaseName": "AppendNotes", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-15", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022610, - "OrganizationUnitId": 7791618, - "FolderKey": "b0c1d2e3-f4a5-4193-8061-728394a5b6c7", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T10:20:02.210Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8", - "Id": 682461109 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/50bc68f07c.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/50bc68f07c.json deleted file mode 100644 index feaca56dc1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/50bc68f07c.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T10:20:02.140Z", - "Message": "Validation Error: Activity 'Append Text' is valid only inside WordApplicationScope.\r\nin Main.xaml\r\n at WordAppendText \"Append Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/6ca6ad5955.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/6ca6ad5955.json deleted file mode 100644 index e7ba31dcc0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/6ca6ad5955.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b0c1d2e3-f4a5-4193-8061-728394a5b6c7", - "Name": "Doc Append", - "Path": "Doc Append", - "Description": "Production folder for document-append processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/README.md deleted file mode 100644 index c0ecd96dca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Append Text Failure - Activity Outside Its Container - -This scenario reproduces an `Append Text` failure where the App-Integration -`Append Text` activity is dropped **outside** a `Word Application Scope` and -faults with `Activity is valid only inside WordApplicationScope` — it has no -file input of its own. - -## What this scenario uncovers - -**Root Cause:** The App-Integration `Append Text` (`WordAppendText`) sits -loose in the sequence with no container. It appends to the document held open -by a surrounding scope; outside one it is invalid, so the job faults -immediately. The file/document/install are not implicated. - -This maps to: -`references/activity-packages/word-activities/playbooks/append-text-missing-container.md` - -The correct agent behavior is to tie the fault to the missing container / -wrong Append Text surface and recommend either nesting it in a -`Word Application Scope` / `Use Word File` or switching to the standalone -`Word Document` `Append Text` (which takes a file path) — not blaming the -file path, document, or install. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a `Word Application Scope`-less `Append Text` loose in the sequence | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's error names the `Append Text` -> activity and the `WordApplicationScope` requirement; the `.xaml` shows the -> activity loose in the `Main Sequence`. Twin of `read-text-missing-container` -> for a different activity. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `append-text-missing-container.md` -- Agent identified the App-Integration Append Text being outside a - `Word Application Scope` as the cause and recommended nesting it in a scope - OR switching to the standalone `Word Document` `Append Text` — without - blaming the file/document/install or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/c422e88123.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/c422e88123.json deleted file mode 100644 index 6c33102a0f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/c422e88123.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-17T10:20:01.380Z", - "Message": "Execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T10:20:02.140Z", - "Message": "Validation Error: Activity 'Append Text' is valid only inside WordApplicationScope.\r\nin Main.xaml\r\n at WordAppendText \"Append Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T10:20:02.190Z", - "Message": "Execution ended in: 00:00:00" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/db3503437c.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/db3503437c.json deleted file mode 100644 index c836e2f1fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/db3503437c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8", - "State": "Faulted", - "ProcessName": "AppendNotes", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-15", - "MachineKey": "", - "StartTime": "2026-06-17T10:20:01.330Z", - "EndTime": "2026-06-17T10:20:02.210Z", - "CreationTime": "2026-06-17T10:20:01.100Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/manifest.json deleted file mode 100644 index a603f1e31f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word append-text missing-container scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6ca6ad5955.json" - }, - { - "match": "or jobs list --folder-key \"b0c1d2e3-f4a5-4193-8061-728394a5b6c7\" --state Faulted --output json", - "file": "db3503437c.json" - }, - { - "match": "or jobs list --folder-key b0c1d2e3-f4a5-4193-8061-728394a5b6c7 --state Faulted --output json", - "file": "db3503437c.json" - }, - { - "match": "or jobs get \"c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8\" --output json", - "file": "0f34c8f658.json" - }, - { - "match": "or jobs get c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8 --output json", - "file": "0f34c8f658.json" - }, - { - "match": "or jobs logs \"c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8\" --level Error --output json", - "file": "50bc68f07c.json" - }, - { - "match": "or jobs logs c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8 --level Error --output json", - "file": "50bc68f07c.json" - }, - { - "match": "or jobs logs \"c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8\" --output json", - "file": "c422e88123.json" - }, - { - "match": "or jobs logs c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8 --output json", - "file": "c422e88123.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/uip-fixture.json new file mode 100644 index 0000000000..01b2c41d43 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b0c1d2e3-f4a5-4193-8061-728394a5b6c7", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AppendNotes\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-15\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T10:20:01.330Z\",\r\n \"EndTime\": \"2026-06-17T10:20:02.210Z\",\r\n \"CreationTime\": \"2026-06-17T10:20:01.100Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T10:20:02.140Z\",\r\n \"Message\": \"Validation Error: Activity 'Append Text' is valid only inside WordApplicationScope.\\r\\nin Main.xaml\\r\\n at WordAppendText \\\"Append Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8\",\r\n \"StartTime\": \"2026-06-17T10:20:01.330Z\",\r\n \"EndTime\": \"2026-06-17T10:20:02.210Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c2d3e4f5-a6b7-4315-9283-94a5b6c7d8e9\",\r\n \"Info\": \"Validation Error: Activity 'Append Text' is valid only inside WordApplicationScope.\\r\\n\\r\\nin Main.xaml\\r\\n at WordAppendText \\\"Append Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Word.Activities.WordException: Activity is valid only inside WordApplicationScope.\",\r\n \"CreationTime\": \"2026-06-17T10:20:01.100Z\",\r\n \"ReleaseName\": \"AppendNotes\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-15\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022610,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b0c1d2e3-f4a5-4193-8061-728394a5b6c7\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T10:20:02.210Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8\",\r\n \"Id\": 682461109\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c1d2e3f4-a5b6-4204-9172-8394a5b6c7d8", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T10:20:01.380Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T10:20:02.140Z\",\r\n \"Message\": \"Validation Error: Activity 'Append Text' is valid only inside WordApplicationScope.\\r\\nin Main.xaml\\r\\n at WordAppendText \\\"Append Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T10:20:02.190Z\",\r\n \"Message\": \"Execution ended in: 00:00:00\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b0c1d2e3-f4a5-4193-8061-728394a5b6c7\",\r\n \"Name\": \"Doc Append\",\r\n \"Path\": \"Doc Append\",\r\n \"Description\": \"Production folder for document-append processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/task.yaml index ad8dfe80da..d705e3eda3 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-missing-container/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/README.md new file mode 100644 index 0000000000..0d42c33a10 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/README.md @@ -0,0 +1,46 @@ +# Append Text Failure - Word Not Installed on the Server (cross-ref) + +This scenario reproduces an `Append Text` COM startup failure on a +locked-down server with no desktop Word: the `Word Application Scope` can't +create the `Word.Application` COM object and faults with `REGDB_E_CLASSNOTREG` +/ "make sure Word application is installed". + +## What this scenario uncovers + +**Root Cause:** App-Integration `Append Text` runs inside a Word Application +Scope (Interop), which needs a registered desktop Word. The server has none, +so the scope faults at startup. "Worked on dev, fails on the server" points +at a host-environment cause. + +This is a **cross-activity validation** of the shared signature: +`references/activity-packages/word-activities/playbooks/word-scope-com-not-installed.md` + +The correct agent behavior is to match the Word-not-installed signature and +recommend either installing desktop Word **or** switching to the standalone +**Word Document `Append Text`** (OpenXML, no Word install) — the +Append-Text-specific, server-friendly path — without fabricating host +actions. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Word Application Scope` + `Append Text` on a server with no Word | + +> **Note on fixtures.** The faulted job's `REGDB_E_CLASSNOTREG` on the +> `Word.Application` CLSID `{000209FF-...}` at scope startup is the same +> signature as the Word Application Scope not-installed case; this scenario +> confirms the agent routes an **Append Text** server failure to that shared +> playbook and gives the Word-Document-activities alternative. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `word-scope-com-not-installed.md` (the shared Word-not-installed signature) +- Agent identified desktop Word missing on the server as the cause and + recommended installing Word **or** switching to the standalone Word + Document Append Text (no Word install) — without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/16cced8fe7.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/16cced8fe7.json deleted file mode 100644 index db38ac4fbf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/16cced8fe7.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112", - "State": "Faulted", - "ProcessName": "ServerAppend", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-17T07:15:02.090Z", - "EndTime": "2026-06-17T07:15:04.180Z", - "CreationTime": "2026-06-17T07:15:01.860Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/5b3cba313f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/5b3cba313f.json deleted file mode 100644 index ba2b64ff68..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/5b3cba313f.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112", - "StartTime": "2026-06-17T07:15:02.090Z", - "EndTime": "2026-06-17T07:15:04.180Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "a6b7c8d9-e0f1-4315-d5b6-c7d8e9f00123", - "Info": "Word Application Scope: Error opening document, make sure Word application is installed. Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\n\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).", - "CreationTime": "2026-06-17T07:15:01.860Z", - "ReleaseName": "ServerAppend", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022830, - "OrganizationUnitId": 7791618, - "FolderKey": "f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T07:15:04.180Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112", - "Id": 682463331 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/6f78deea10.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/6f78deea10.json deleted file mode 100644 index a74790c665..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/6f78deea10.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001", - "Name": "Server Docs", - "Path": "Server Docs", - "Description": "Production folder for server-side document automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/99010efbc3.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/99010efbc3.json deleted file mode 100644 index cad7b14a5a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/99010efbc3.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-17T07:15:02.140Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T07:15:03.520Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T07:15:04.110Z", - "Message": "Error opening document, make sure Word application is installed. Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T07:15:04.160Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/README.md deleted file mode 100644 index 85b4750784..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Append Text Failure - Word Not Installed on the Server (cross-ref) - -This scenario reproduces an `Append Text` COM startup failure on a -locked-down server with no desktop Word: the `Word Application Scope` can't -create the `Word.Application` COM object and faults with `REGDB_E_CLASSNOTREG` -/ "make sure Word application is installed". - -## What this scenario uncovers - -**Root Cause:** App-Integration `Append Text` runs inside a Word Application -Scope (Interop), which needs a registered desktop Word. The server has none, -so the scope faults at startup. "Worked on dev, fails on the server" points -at a host-environment cause. - -This is a **cross-activity validation** of the shared signature: -`references/activity-packages/word-activities/playbooks/word-scope-com-not-installed.md` - -The correct agent behavior is to match the Word-not-installed signature and -recommend either installing desktop Word **or** switching to the standalone -**Word Document `Append Text`** (OpenXML, no Word install) — the -Append-Text-specific, server-friendly path — without fabricating host -actions. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Word Application Scope` + `Append Text` on a server with no Word | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's `REGDB_E_CLASSNOTREG` on the -> `Word.Application` CLSID `{000209FF-...}` at scope startup is the same -> signature as the Word Application Scope not-installed case; this scenario -> confirms the agent routes an **Append Text** server failure to that shared -> playbook and gives the Word-Document-activities alternative. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `word-scope-com-not-installed.md` (the shared Word-not-installed signature) -- Agent identified desktop Word missing on the server as the cause and - recommended installing Word **or** switching to the standalone Word - Document Append Text (no Word install) — without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/ebae7a7ad4.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/ebae7a7ad4.json deleted file mode 100644 index e944901efb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/ebae7a7ad4.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T07:15:04.110Z", - "Message": "Error opening document, make sure Word application is installed. Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/manifest.json deleted file mode 100644 index 1b60604357..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word append-text not-installed scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6f78deea10.json" - }, - { - "match": "or jobs list --folder-key \"f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001\" --state Faulted --output json", - "file": "16cced8fe7.json" - }, - { - "match": "or jobs list --folder-key f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001 --state Faulted --output json", - "file": "16cced8fe7.json" - }, - { - "match": "or jobs get \"a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112\" --output json", - "file": "5b3cba313f.json" - }, - { - "match": "or jobs get a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112 --output json", - "file": "5b3cba313f.json" - }, - { - "match": "or jobs logs \"a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112\" --level Error --output json", - "file": "ebae7a7ad4.json" - }, - { - "match": "or jobs logs a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112 --level Error --output json", - "file": "ebae7a7ad4.json" - }, - { - "match": "or jobs logs \"a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112\" --output json", - "file": "99010efbc3.json" - }, - { - "match": "or jobs logs a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112 --output json", - "file": "99010efbc3.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/uip-fixture.json new file mode 100644 index 0000000000..39d27b8290 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ServerAppend\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T07:15:02.090Z\",\r\n \"EndTime\": \"2026-06-17T07:15:04.180Z\",\r\n \"CreationTime\": \"2026-06-17T07:15:01.860Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T07:15:04.110Z\",\r\n \"Message\": \"Error opening document, make sure Word application is installed. Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112\",\r\n \"StartTime\": \"2026-06-17T07:15:02.090Z\",\r\n \"EndTime\": \"2026-06-17T07:15:04.180Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"a6b7c8d9-e0f1-4315-d5b6-c7d8e9f00123\",\r\n \"Info\": \"Word Application Scope: Error opening document, make sure Word application is installed. Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\n\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\",\r\n \"CreationTime\": \"2026-06-17T07:15:01.860Z\",\r\n \"ReleaseName\": \"ServerAppend\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022830,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T07:15:04.180Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112\",\r\n \"Id\": 682463331\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a5b6c7d8-e9f0-4204-d5b6-c7d8e9f00112", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T07:15:02.140Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T07:15:03.520Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T07:15:04.110Z\",\r\n \"Message\": \"Error opening document, make sure Word application is installed. Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T07:15:04.160Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f4a5b6c7-d8e9-4193-c4a5-b6c7d8e9f001\",\r\n \"Name\": \"Server Docs\",\r\n \"Path\": \"Server Docs\",\r\n \"Description\": \"Production folder for server-side document automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/task.yaml index ecd291447c..8383d9a8ad 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-not-installed/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/README.md new file mode 100644 index 0000000000..7c16ec7b30 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/README.md @@ -0,0 +1,44 @@ +# Append Text Failure - "Archive file cannot be size zero" + +This scenario reproduces an `Append Text` failure where the target `.docx` +is a **0-byte file** (not a valid OpenXML package), so opening it faults with +`System.IO.InvalidDataException: Archive file cannot be size zero`. + +## What this scenario uncovers + +**Root Cause:** A `.docx` is an OpenXML ZIP archive; a 0-byte file (a renamed +`.txt`, or a failed/truncated write) has no archive structure, so the Word +scope rejects it on open — before `Append Text` runs. The extension alone +doesn't make a valid Word document. + +This maps to: +`references/activity-packages/word-activities/playbooks/append-text-zero-byte-file.md` + +The correct agent behavior is to tie the error to the 0-byte / invalid input +file and recommend deleting it + `Create if not exists` (or fixing the +upstream that produced it) — not blaming a missing Word install, a lock, the +activity placement, or the workflow logic. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Word Application Scope` opening `data\Notes.docx` + `Append Text` inside | + +> **Note on fixtures.** The faulted job's `InvalidDataException: Archive file +> cannot be size zero` fires at the scope's document open (before Append +> Text), and the activity placement is valid — isolating the cause to the +> 0-byte input file. Distinct from the corrupt-but-non-empty case +> (`word-scope-file-corrupted`). + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `append-text-zero-byte-file.md` +- Agent identified the 0-byte / invalid `.docx` as the cause and recommended + deleting it + `Create if not exists` (and/or fixing the upstream write) — + without blaming install/lock/placement/logic or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/4519cfade3.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/4519cfade3.json deleted file mode 100644 index 8b283baa2a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/4519cfade3.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9", - "Name": "Report Build", - "Path": "Report Build", - "Description": "Production folder for report assembly processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/4c8fb557ac.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/4c8fb557ac.json deleted file mode 100644 index e7f39f5c6e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/4c8fb557ac.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-17T11:05:02.460Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T11:05:03.980Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T11:05:05.190Z", - "Message": "Error opening document 'C:\\UiPath\\Report Build\\data\\Notes.docx'. Archive file cannot be size zero.\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T11:05:05.240Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/793d617b10.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/793d617b10.json deleted file mode 100644 index edf15b7b08..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/793d617b10.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0", - "StartTime": "2026-06-17T11:05:02.410Z", - "EndTime": "2026-06-17T11:05:05.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "e4f5a6b7-c8d9-4315-c4a5-b6c7d8e9f001", - "Info": "Word Application Scope: Error opening document 'C:\\UiPath\\Report Build\\data\\Notes.docx'. Archive file cannot be size zero.\r\n\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.InvalidDataException: Archive file cannot be size zero.", - "CreationTime": "2026-06-17T11:05:02.180Z", - "ReleaseName": "ReportAppender", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-16", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022720, - "OrganizationUnitId": 7791618, - "FolderKey": "d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T11:05:05.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0", - "Id": 682462220 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/90fb3d3a43.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/90fb3d3a43.json deleted file mode 100644 index e672510272..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/90fb3d3a43.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0", - "State": "Faulted", - "ProcessName": "ReportAppender", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-16", - "MachineKey": "", - "StartTime": "2026-06-17T11:05:02.410Z", - "EndTime": "2026-06-17T11:05:05.260Z", - "CreationTime": "2026-06-17T11:05:02.180Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/README.md deleted file mode 100644 index 0e82bae541..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Append Text Failure - "Archive file cannot be size zero" - -This scenario reproduces an `Append Text` failure where the target `.docx` -is a **0-byte file** (not a valid OpenXML package), so opening it faults with -`System.IO.InvalidDataException: Archive file cannot be size zero`. - -## What this scenario uncovers - -**Root Cause:** A `.docx` is an OpenXML ZIP archive; a 0-byte file (a renamed -`.txt`, or a failed/truncated write) has no archive structure, so the Word -scope rejects it on open — before `Append Text` runs. The extension alone -doesn't make a valid Word document. - -This maps to: -`references/activity-packages/word-activities/playbooks/append-text-zero-byte-file.md` - -The correct agent behavior is to tie the error to the 0-byte / invalid input -file and recommend deleting it + `Create if not exists` (or fixing the -upstream that produced it) — not blaming a missing Word install, a lock, the -activity placement, or the workflow logic. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Word Application Scope` opening `data\Notes.docx` + `Append Text` inside | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's `InvalidDataException: Archive file -> cannot be size zero` fires at the scope's document open (before Append -> Text), and the activity placement is valid — isolating the cause to the -> 0-byte input file. Distinct from the corrupt-but-non-empty case -> (`word-scope-file-corrupted`). - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `append-text-zero-byte-file.md` -- Agent identified the 0-byte / invalid `.docx` as the cause and recommended - deleting it + `Create if not exists` (and/or fixing the upstream write) — - without blaming install/lock/placement/logic or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/c28d4b82ef.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/c28d4b82ef.json deleted file mode 100644 index d38d9182c2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/c28d4b82ef.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T11:05:05.190Z", - "Message": "Error opening document 'C:\\UiPath\\Report Build\\data\\Notes.docx'. Archive file cannot be size zero.\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/manifest.json deleted file mode 100644 index abbe2949df..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word append-text zero-byte-file scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "4519cfade3.json" - }, - { - "match": "or jobs list --folder-key \"d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9\" --state Faulted --output json", - "file": "90fb3d3a43.json" - }, - { - "match": "or jobs list --folder-key d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9 --state Faulted --output json", - "file": "90fb3d3a43.json" - }, - { - "match": "or jobs get \"e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0\" --output json", - "file": "793d617b10.json" - }, - { - "match": "or jobs get e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0 --output json", - "file": "793d617b10.json" - }, - { - "match": "or jobs logs \"e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0\" --level Error --output json", - "file": "c28d4b82ef.json" - }, - { - "match": "or jobs logs e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0 --level Error --output json", - "file": "c28d4b82ef.json" - }, - { - "match": "or jobs logs \"e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0\" --output json", - "file": "4c8fb557ac.json" - }, - { - "match": "or jobs logs e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0 --output json", - "file": "4c8fb557ac.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/uip-fixture.json new file mode 100644 index 0000000000..c5e35b2f16 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportAppender\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-16\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T11:05:02.410Z\",\r\n \"EndTime\": \"2026-06-17T11:05:05.260Z\",\r\n \"CreationTime\": \"2026-06-17T11:05:02.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T11:05:05.190Z\",\r\n \"Message\": \"Error opening document 'C:\\\\UiPath\\\\Report Build\\\\data\\\\Notes.docx'. Archive file cannot be size zero.\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0\",\r\n \"StartTime\": \"2026-06-17T11:05:02.410Z\",\r\n \"EndTime\": \"2026-06-17T11:05:05.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"e4f5a6b7-c8d9-4315-c4a5-b6c7d8e9f001\",\r\n \"Info\": \"Word Application Scope: Error opening document 'C:\\\\UiPath\\\\Report Build\\\\data\\\\Notes.docx'. Archive file cannot be size zero.\\r\\n\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.InvalidDataException: Archive file cannot be size zero.\",\r\n \"CreationTime\": \"2026-06-17T11:05:02.180Z\",\r\n \"ReleaseName\": \"ReportAppender\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-16\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022720,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T11:05:05.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0\",\r\n \"Id\": 682462220\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e3f4a5b6-c7d8-4204-b394-a5b6c7d8e9f0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T11:05:02.460Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T11:05:03.980Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T11:05:05.190Z\",\r\n \"Message\": \"Error opening document 'C:\\\\UiPath\\\\Report Build\\\\data\\\\Notes.docx'. Archive file cannot be size zero.\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T11:05:05.240Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d2e3f4a5-b6c7-4193-a283-94a5b6c7d8e9\",\r\n \"Name\": \"Report Build\",\r\n \"Path\": \"Report Build\",\r\n \"Description\": \"Production folder for report assembly processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/task.yaml index 4b2142a766..bbc18e5454 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-append-text-zero-byte-file/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/README.md new file mode 100644 index 0000000000..d110423860 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/README.md @@ -0,0 +1,47 @@ +# Export to PDF Failure - COM Interop Hang / Crash (Orphaned WINWORD) + +This scenario reproduces an intermittent `Export to PDF` COM failure where an +orphaned `WINWORD.EXE` (a Word instance already running on the host) blocks +the export's COM call, which faults with `COMException` (`RPC_E_CALL_REJECTED` +/ `0x80010001`). + +## What this scenario uncovers + +**Root Cause:** A stale/orphaned `WINWORD.EXE` blocks the export call; the +job log warns "A Microsoft Word instance was already running on the host". +The intermittency ("re-running sometimes works") is the tell of a transient +busy/locked Word state — not a code defect, not the output path (which is +valid here). + +This maps to: +`references/activity-packages/word-activities/playbooks/export-pdf-com-hang.md` + +The correct agent behavior is to tie the `COMException` to a busy/orphaned +WINWORD and recommend clearing it (Kill Process WINWORD before the scope + +dispose, ensure input free), optionally the Invoke Code C# `ExportAsFixedFormat` +fallback — framed as host/workflow steps for the off-host user, not blaming +the output path or document. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Word Application Scope` + `Export to PDF` to a valid `.pdf` path | + +> **Note on fixtures.** The full job log includes a "Word already running on +> the host" warning immediately before the `RPC_E_CALL_REJECTED` COM error, +> and the output path/folder is valid — isolating the cause to the busy/ +> orphaned WINWORD rather than the missing-dir or path-format scenarios. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `export-pdf-com-hang.md` +- Agent identified a busy/orphaned WINWORD blocking the export's COM call as + the cause and recommended clearing it (Kill Process WINWORD before the + scope / dispose / ensure input free, optionally the Invoke Code fallback) — + without fabricating host actions or blaming the output path/document + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/4ba50747c8.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/4ba50747c8.json deleted file mode 100644 index 98d692e0ca..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/4ba50747c8.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e7f8a9b0-c1d2-4193-de4f-5061728394a5", - "Name": "Batch PDF", - "Path": "Batch PDF", - "Description": "Production folder for batch PDF conversion.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/8180738813.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/8180738813.json deleted file mode 100644 index df077d32ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/8180738813.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T06:00:09.810Z", - "Message": "Export to PDF: The call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/README.md deleted file mode 100644 index 07ebf7ab96..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Export to PDF Failure - COM Interop Hang / Crash (Orphaned WINWORD) - -This scenario reproduces an intermittent `Export to PDF` COM failure where an -orphaned `WINWORD.EXE` (a Word instance already running on the host) blocks -the export's COM call, which faults with `COMException` (`RPC_E_CALL_REJECTED` -/ `0x80010001`). - -## What this scenario uncovers - -**Root Cause:** A stale/orphaned `WINWORD.EXE` blocks the export call; the -job log warns "A Microsoft Word instance was already running on the host". -The intermittency ("re-running sometimes works") is the tell of a transient -busy/locked Word state — not a code defect, not the output path (which is -valid here). - -This maps to: -`references/activity-packages/word-activities/playbooks/export-pdf-com-hang.md` - -The correct agent behavior is to tie the `COMException` to a busy/orphaned -WINWORD and recommend clearing it (Kill Process WINWORD before the scope + -dispose, ensure input free), optionally the Invoke Code C# `ExportAsFixedFormat` -fallback — framed as host/workflow steps for the off-host user, not blaming -the output path or document. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Word Application Scope` + `Export to PDF` to a valid `.pdf` path | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The full job log includes a "Word already running on -> the host" warning immediately before the `RPC_E_CALL_REJECTED` COM error, -> and the output path/folder is valid — isolating the cause to the busy/ -> orphaned WINWORD rather than the missing-dir or path-format scenarios. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `export-pdf-com-hang.md` -- Agent identified a busy/orphaned WINWORD blocking the export's COM call as - the cause and recommended clearing it (Kill Process WINWORD before the - scope / dispose / ensure input free, optionally the Invoke Code fallback) — - without fabricating host actions or blaming the output path/document diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/b654210dee.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/b654210dee.json deleted file mode 100644 index 8bdf3ab9ee..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/b654210dee.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-17T06:00:03.460Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T06:00:05.020Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-17T06:00:06.340Z", - "Message": "[Word Application Scope] A Microsoft Word instance was already running on the host when the scope started." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T06:00:07.610Z", - "Message": "[Word Application Scope] Document 'data\\Contract.docx' opened" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T06:00:09.810Z", - "Message": "Export to PDF: The call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T06:00:09.860Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/be7145ebf8.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/be7145ebf8.json deleted file mode 100644 index ed765870df..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/be7145ebf8.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f8a9b0c1-d2e3-4204-ef50-61728394a5b6", - "State": "Faulted", - "ProcessName": "BatchPdfConvert", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-17T06:00:03.410Z", - "EndTime": "2026-06-17T06:00:09.880Z", - "CreationTime": "2026-06-17T06:00:03.180Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/e773619b2a.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/e773619b2a.json deleted file mode 100644 index 0cdba51c47..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/e773619b2a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f8a9b0c1-d2e3-4204-ef50-61728394a5b6", - "StartTime": "2026-06-17T06:00:03.410Z", - "EndTime": "2026-06-17T06:00:09.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "f9a0b1c2-d3e4-4315-ef50-61728394a5c7", - "Info": "Export to PDF: Retrieving the COM class factory failed / the call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\r\n\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))", - "CreationTime": "2026-06-17T06:00:03.180Z", - "ReleaseName": "BatchPdfConvert", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022500, - "OrganizationUnitId": 7791618, - "FolderKey": "e7f8a9b0-c1d2-4193-de4f-5061728394a5", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T06:00:09.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f8a9b0c1-d2e3-4204-ef50-61728394a5b6", - "Id": 682460998 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/manifest.json deleted file mode 100644 index 166d1e156e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word export-pdf com-hang scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "4ba50747c8.json" - }, - { - "match": "or jobs list --folder-key \"e7f8a9b0-c1d2-4193-de4f-5061728394a5\" --state Faulted --output json", - "file": "be7145ebf8.json" - }, - { - "match": "or jobs list --folder-key e7f8a9b0-c1d2-4193-de4f-5061728394a5 --state Faulted --output json", - "file": "be7145ebf8.json" - }, - { - "match": "or jobs get \"f8a9b0c1-d2e3-4204-ef50-61728394a5b6\" --output json", - "file": "e773619b2a.json" - }, - { - "match": "or jobs get f8a9b0c1-d2e3-4204-ef50-61728394a5b6 --output json", - "file": "e773619b2a.json" - }, - { - "match": "or jobs logs \"f8a9b0c1-d2e3-4204-ef50-61728394a5b6\" --level Error --output json", - "file": "8180738813.json" - }, - { - "match": "or jobs logs f8a9b0c1-d2e3-4204-ef50-61728394a5b6 --level Error --output json", - "file": "8180738813.json" - }, - { - "match": "or jobs logs \"f8a9b0c1-d2e3-4204-ef50-61728394a5b6\" --output json", - "file": "b654210dee.json" - }, - { - "match": "or jobs logs f8a9b0c1-d2e3-4204-ef50-61728394a5b6 --output json", - "file": "b654210dee.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/uip-fixture.json new file mode 100644 index 0000000000..8e1325f964 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e7f8a9b0-c1d2-4193-de4f-5061728394a5", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f8a9b0c1-d2e3-4204-ef50-61728394a5b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BatchPdfConvert\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T06:00:03.410Z\",\r\n \"EndTime\": \"2026-06-17T06:00:09.880Z\",\r\n \"CreationTime\": \"2026-06-17T06:00:03.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f8a9b0c1-d2e3-4204-ef50-61728394a5b6", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T06:00:09.810Z\",\r\n \"Message\": \"Export to PDF: The call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f8a9b0c1-d2e3-4204-ef50-61728394a5b6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f8a9b0c1-d2e3-4204-ef50-61728394a5b6\",\r\n \"StartTime\": \"2026-06-17T06:00:03.410Z\",\r\n \"EndTime\": \"2026-06-17T06:00:09.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"f9a0b1c2-d3e4-4315-ef50-61728394a5c7\",\r\n \"Info\": \"Export to PDF: Retrieving the COM class factory failed / the call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\\r\\n\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\",\r\n \"CreationTime\": \"2026-06-17T06:00:03.180Z\",\r\n \"ReleaseName\": \"BatchPdfConvert\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022500,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"e7f8a9b0-c1d2-4193-de4f-5061728394a5\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T06:00:09.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f8a9b0c1-d2e3-4204-ef50-61728394a5b6\",\r\n \"Id\": 682460998\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f8a9b0c1-d2e3-4204-ef50-61728394a5b6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T06:00:03.460Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T06:00:05.020Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-17T06:00:06.340Z\",\r\n \"Message\": \"[Word Application Scope] A Microsoft Word instance was already running on the host when the scope started.\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T06:00:07.610Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\Contract.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T06:00:09.810Z\",\r\n \"Message\": \"Export to PDF: The call was rejected by callee. (Exception from HRESULT: 0x80010001 (RPC_E_CALL_REJECTED))\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T06:00:09.860Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e7f8a9b0-c1d2-4193-de4f-5061728394a5\",\r\n \"Name\": \"Batch PDF\",\r\n \"Path\": \"Batch PDF\",\r\n \"Description\": \"Production folder for batch PDF conversion.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/task.yaml index a8725a4b6f..c0dc4e8274 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-com-hang/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/README.md new file mode 100644 index 0000000000..63072c0aa3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/README.md @@ -0,0 +1,45 @@ +# Export to PDF Failure - Generic "Command Failed" (Output Directory Missing) + +This scenario reproduces an `Export to PDF` failure that surfaces only as a +generic `Command Failed` because the output directory +(`C:\Output\2026\Invoices\`) does not exist and the activity does not +auto-create it (no `Create Folder` precedes the export). + +## What this scenario uncovers + +**Root Cause:** `WordExportToPdf` writes to a nested dated folder that isn't +created; the activity won't build the directory, so the save fails and +reports only `Command Failed`. The input document opens fine — the failure +is on the write side. + +This maps to: +`references/activity-packages/word-activities/playbooks/export-pdf-missing-output-dir.md` + +The correct agent behavior is to tie the generic error to the missing output +directory (via the XAML output path + the absence of a `Create Folder`) and +recommend adding a `Create Folder` before the export — not blaming Word/COM, +the input document, or a missing install. This tests disambiguating a +detail-free `Command Failed`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Word Application Scope` + `Export to PDF` to `C:\Output\2026\Invoices\Invoice.pdf`, no Create Folder | + +> **Note on fixtures.** The job log shows the input opening (`Document +> 'data\\Invoice.docx' opened`) before `Export to PDF: Command Failed`, +> locating the fault at the write; the `.xaml` shows the nested output path +> and no `Create Folder`. The error itself is intentionally detail-free. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `export-pdf-missing-output-dir.md` +- Agent identified the non-existent output directory (activity won't + auto-create) as the cause and recommended a `Create Folder` before the + export — without blaming Word/COM/document or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/26a55dbd49.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/26a55dbd49.json deleted file mode 100644 index 3958c508bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/26a55dbd49.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-17T08:30:02.160Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T08:30:03.880Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T08:30:05.220Z", - "Message": "[Word Application Scope] Document 'data\\Invoice.docx' opened" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T08:30:06.470Z", - "Message": "Export to PDF: Command Failed\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T08:30:06.520Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/567df0cf7d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/567df0cf7d.json deleted file mode 100644 index 392119dc88..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/567df0cf7d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172", - "StartTime": "2026-06-17T08:30:02.110Z", - "EndTime": "2026-06-17T08:30:06.540Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b5c6d7e8-f9a0-4315-bc2d-3e4f50617283", - "Info": "Export to PDF: Command Failed\r\n\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Word.Activities.WordException: Command Failed", - "CreationTime": "2026-06-17T08:30:01.890Z", - "ReleaseName": "InvoicePdfExport", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-13", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022280, - "OrganizationUnitId": 7791618, - "FolderKey": "a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T08:30:06.540Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172", - "Id": 682458776 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/6c791a8ea7.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/6c791a8ea7.json deleted file mode 100644 index ad8a142b47..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/6c791a8ea7.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172", - "State": "Faulted", - "ProcessName": "InvoicePdfExport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-13", - "MachineKey": "", - "StartTime": "2026-06-17T08:30:02.110Z", - "EndTime": "2026-06-17T08:30:06.540Z", - "CreationTime": "2026-06-17T08:30:01.890Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/README.md deleted file mode 100644 index 8dc65e7a1b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Export to PDF Failure - Generic "Command Failed" (Output Directory Missing) - -This scenario reproduces an `Export to PDF` failure that surfaces only as a -generic `Command Failed` because the output directory -(`C:\Output\2026\Invoices\`) does not exist and the activity does not -auto-create it (no `Create Folder` precedes the export). - -## What this scenario uncovers - -**Root Cause:** `WordExportToPdf` writes to a nested dated folder that isn't -created; the activity won't build the directory, so the save fails and -reports only `Command Failed`. The input document opens fine — the failure -is on the write side. - -This maps to: -`references/activity-packages/word-activities/playbooks/export-pdf-missing-output-dir.md` - -The correct agent behavior is to tie the generic error to the missing output -directory (via the XAML output path + the absence of a `Create Folder`) and -recommend adding a `Create Folder` before the export — not blaming Word/COM, -the input document, or a missing install. This tests disambiguating a -detail-free `Command Failed`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Word Application Scope` + `Export to PDF` to `C:\Output\2026\Invoices\Invoice.pdf`, no Create Folder | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The job log shows the input opening (`Document -> 'data\\Invoice.docx' opened`) before `Export to PDF: Command Failed`, -> locating the fault at the write; the `.xaml` shows the nested output path -> and no `Create Folder`. The error itself is intentionally detail-free. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `export-pdf-missing-output-dir.md` -- Agent identified the non-existent output directory (activity won't - auto-create) as the cause and recommended a `Create Folder` before the - export — without blaming Word/COM/document or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/b590f583da.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/b590f583da.json deleted file mode 100644 index dbee658b89..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/b590f583da.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061", - "Name": "PDF Export", - "Path": "PDF Export", - "Description": "Production folder for PDF export processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/db7f03e276.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/db7f03e276.json deleted file mode 100644 index 7476bb400a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/db7f03e276.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T08:30:06.470Z", - "Message": "Export to PDF: Command Failed\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/manifest.json deleted file mode 100644 index 151dd6060f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word export-pdf missing-output-dir scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "b590f583da.json" - }, - { - "match": "or jobs list --folder-key \"a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061\" --state Faulted --output json", - "file": "6c791a8ea7.json" - }, - { - "match": "or jobs list --folder-key a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061 --state Faulted --output json", - "file": "6c791a8ea7.json" - }, - { - "match": "or jobs get \"b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172\" --output json", - "file": "567df0cf7d.json" - }, - { - "match": "or jobs get b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172 --output json", - "file": "567df0cf7d.json" - }, - { - "match": "or jobs logs \"b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172\" --level Error --output json", - "file": "db7f03e276.json" - }, - { - "match": "or jobs logs b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172 --level Error --output json", - "file": "db7f03e276.json" - }, - { - "match": "or jobs logs \"b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172\" --output json", - "file": "26a55dbd49.json" - }, - { - "match": "or jobs logs b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172 --output json", - "file": "26a55dbd49.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/uip-fixture.json new file mode 100644 index 0000000000..913ec448b0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoicePdfExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-13\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T08:30:02.110Z\",\r\n \"EndTime\": \"2026-06-17T08:30:06.540Z\",\r\n \"CreationTime\": \"2026-06-17T08:30:01.890Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T08:30:06.470Z\",\r\n \"Message\": \"Export to PDF: Command Failed\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172\",\r\n \"StartTime\": \"2026-06-17T08:30:02.110Z\",\r\n \"EndTime\": \"2026-06-17T08:30:06.540Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b5c6d7e8-f9a0-4315-bc2d-3e4f50617283\",\r\n \"Info\": \"Export to PDF: Command Failed\\r\\n\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Word.Activities.WordException: Command Failed\",\r\n \"CreationTime\": \"2026-06-17T08:30:01.890Z\",\r\n \"ReleaseName\": \"InvoicePdfExport\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-13\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022280,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T08:30:06.540Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172\",\r\n \"Id\": 682458776\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b4c5d6e7-f8a9-4204-ab1c-2d3e4f506172", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T08:30:02.160Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T08:30:03.880Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T08:30:05.220Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\Invoice.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T08:30:06.470Z\",\r\n \"Message\": \"Export to PDF: Command Failed\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T08:30:06.520Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a3b4c5d6-e7f8-4193-9a0b-1c2d3e4f5061\",\r\n \"Name\": \"PDF Export\",\r\n \"Path\": \"PDF Export\",\r\n \"Description\": \"Production folder for PDF export processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/task.yaml index 4725c6a432..7f36974bc5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-missing-output-dir/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/README.md new file mode 100644 index 0000000000..f2361a67c5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/README.md @@ -0,0 +1,45 @@ +# Export to PDF Failure - Malformed Output Path / Missing .pdf + +This scenario reproduces an `Export to PDF` failure where the output path is +built by string concatenation **without a `.pdf` extension** +(`outFolder + "\" + reportName` → `C:\Reports\MonthlyReport`), so the +activity throws `ArgumentException: The export file path must specify a .pdf +file`. + +## What this scenario uncovers + +**Root Cause:** `WordExportToPdf` requires the output to be a file ending in +`.pdf`. The concatenated path omits the extension, so the resolved value is +an invalid PDF target. The document opens fine — the failure is purely the +malformed output path. + +This maps to: +`references/activity-packages/word-activities/playbooks/export-pdf-output-path-format.md` + +The correct agent behavior is to tie the fault to the missing `.pdf` +extension / path formatting (via the XAML expression + the error's `Value +was 'C:\Reports\MonthlyReport'`) and recommend `Path.Combine(folder, name & +".pdf")` — not blaming a missing folder, COM, or the document. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Export to PDF` `FileName = outFolder + "\" + reportName` (no `.pdf`) | + +> **Note on fixtures.** The faulted job's `ArgumentException` names the exact +> malformed value (`C:\Reports\MonthlyReport`), matching the XAML expression; +> the input opens first, locating the fault at the output path. Distinct from +> the generic `Command Failed` of the missing-output-directory scenario. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `export-pdf-output-path-format.md` +- Agent identified the output path lacking the `.pdf` extension as the cause + and recommended a `.pdf`-suffixed path (`Path.Combine` / append `".pdf"`) — + without blaming a missing folder/COM/document or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/225e0339ec.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/225e0339ec.json deleted file mode 100644 index ca15188e00..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/225e0339ec.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-17T09:40:06.630Z", - "Message": "Export to PDF: The export file path must specify a .pdf file. Value was 'C:\\Reports\\MonthlyReport'.\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/409059be85.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/409059be85.json deleted file mode 100644 index 31119bc3c5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/409059be85.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d6e7f8a9-b0c1-4204-cd3e-4f5061728394", - "State": "Faulted", - "ProcessName": "ReportPdfGen", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-14", - "MachineKey": "", - "StartTime": "2026-06-17T09:40:02.220Z", - "EndTime": "2026-06-17T09:40:06.700Z", - "CreationTime": "2026-06-17T09:40:01.990Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/6dd09c93f0.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/6dd09c93f0.json deleted file mode 100644 index 50a95c34cb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/6dd09c93f0.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d6e7f8a9-b0c1-4204-cd3e-4f5061728394", - "StartTime": "2026-06-17T09:40:02.220Z", - "EndTime": "2026-06-17T09:40:06.700Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "d7e8f9a0-b1c2-4315-cd3e-4f5061728395", - "Info": "Export to PDF: Could not export the document to 'C:\\Reports\\MonthlyReport'. The output path is not a valid PDF file path (expected a file name ending in .pdf).\r\n\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: The export file path must specify a .pdf file. Value was 'C:\\Reports\\MonthlyReport'.", - "CreationTime": "2026-06-17T09:40:01.990Z", - "ReleaseName": "ReportPdfGen", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-14", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022390, - "OrganizationUnitId": 7791618, - "FolderKey": "c5d6e7f8-a9b0-4193-bc2d-3e4f50617283", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-17T09:40:06.700Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d6e7f8a9-b0c1-4204-cd3e-4f5061728394", - "Id": 682459887 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/README.md deleted file mode 100644 index 9eab39d60f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Export to PDF Failure - Malformed Output Path / Missing .pdf - -This scenario reproduces an `Export to PDF` failure where the output path is -built by string concatenation **without a `.pdf` extension** -(`outFolder + "\" + reportName` → `C:\Reports\MonthlyReport`), so the -activity throws `ArgumentException: The export file path must specify a .pdf -file`. - -## What this scenario uncovers - -**Root Cause:** `WordExportToPdf` requires the output to be a file ending in -`.pdf`. The concatenated path omits the extension, so the resolved value is -an invalid PDF target. The document opens fine — the failure is purely the -malformed output path. - -This maps to: -`references/activity-packages/word-activities/playbooks/export-pdf-output-path-format.md` - -The correct agent behavior is to tie the fault to the missing `.pdf` -extension / path formatting (via the XAML expression + the error's `Value -was 'C:\Reports\MonthlyReport'`) and recommend `Path.Combine(folder, name & -".pdf")` — not blaming a missing folder, COM, or the document. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Export to PDF` `FileName = outFolder + "\" + reportName` (no `.pdf`) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's `ArgumentException` names the exact -> malformed value (`C:\Reports\MonthlyReport`), matching the XAML expression; -> the input opens first, locating the fault at the output path. Distinct from -> the generic `Command Failed` of the missing-output-directory scenario. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `export-pdf-output-path-format.md` -- Agent identified the output path lacking the `.pdf` extension as the cause - and recommended a `.pdf`-suffixed path (`Path.Combine` / append `".pdf"`) — - without blaming a missing folder/COM/document or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/d42ac4417c.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/d42ac4417c.json deleted file mode 100644 index 660f946bb1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/d42ac4417c.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-17T09:40:02.270Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T09:40:03.980Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-17T09:40:05.310Z", - "Message": "[Word Application Scope] Document 'data\\Report.docx' opened" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-17T09:40:06.630Z", - "Message": "Export to PDF: The export file path must specify a .pdf file. Value was 'C:\\Reports\\MonthlyReport'.\r\nin Main.xaml\r\n at WordExportToPdf \"Export to PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-17T09:40:06.680Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/f0ab3ecf69.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/f0ab3ecf69.json deleted file mode 100644 index 73dbb989b5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/f0ab3ecf69.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c5d6e7f8-a9b0-4193-bc2d-3e4f50617283", - "Name": "Report PDFs", - "Path": "Report PDFs", - "Description": "Production folder for report PDF generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/manifest.json deleted file mode 100644 index d6c2ec369d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the export-pdf output-path-format scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "f0ab3ecf69.json" - }, - { - "match": "or jobs list --folder-key \"c5d6e7f8-a9b0-4193-bc2d-3e4f50617283\" --state Faulted --output json", - "file": "409059be85.json" - }, - { - "match": "or jobs list --folder-key c5d6e7f8-a9b0-4193-bc2d-3e4f50617283 --state Faulted --output json", - "file": "409059be85.json" - }, - { - "match": "or jobs get \"d6e7f8a9-b0c1-4204-cd3e-4f5061728394\" --output json", - "file": "6dd09c93f0.json" - }, - { - "match": "or jobs get d6e7f8a9-b0c1-4204-cd3e-4f5061728394 --output json", - "file": "6dd09c93f0.json" - }, - { - "match": "or jobs logs \"d6e7f8a9-b0c1-4204-cd3e-4f5061728394\" --level Error --output json", - "file": "225e0339ec.json" - }, - { - "match": "or jobs logs d6e7f8a9-b0c1-4204-cd3e-4f5061728394 --level Error --output json", - "file": "225e0339ec.json" - }, - { - "match": "or jobs logs \"d6e7f8a9-b0c1-4204-cd3e-4f5061728394\" --output json", - "file": "d42ac4417c.json" - }, - { - "match": "or jobs logs d6e7f8a9-b0c1-4204-cd3e-4f5061728394 --output json", - "file": "d42ac4417c.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/uip-fixture.json new file mode 100644 index 0000000000..997b9c6a3c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c5d6e7f8-a9b0-4193-bc2d-3e4f50617283", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d6e7f8a9-b0c1-4204-cd3e-4f5061728394\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportPdfGen\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-14\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-17T09:40:02.220Z\",\r\n \"EndTime\": \"2026-06-17T09:40:06.700Z\",\r\n \"CreationTime\": \"2026-06-17T09:40:01.990Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d6e7f8a9-b0c1-4204-cd3e-4f5061728394", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T09:40:06.630Z\",\r\n \"Message\": \"Export to PDF: The export file path must specify a .pdf file. Value was 'C:\\\\Reports\\\\MonthlyReport'.\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d6e7f8a9-b0c1-4204-cd3e-4f5061728394", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d6e7f8a9-b0c1-4204-cd3e-4f5061728394\",\r\n \"StartTime\": \"2026-06-17T09:40:02.220Z\",\r\n \"EndTime\": \"2026-06-17T09:40:06.700Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"d7e8f9a0-b1c2-4315-cd3e-4f5061728395\",\r\n \"Info\": \"Export to PDF: Could not export the document to 'C:\\\\Reports\\\\MonthlyReport'. The output path is not a valid PDF file path (expected a file name ending in .pdf).\\r\\n\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: The export file path must specify a .pdf file. Value was 'C:\\\\Reports\\\\MonthlyReport'.\",\r\n \"CreationTime\": \"2026-06-17T09:40:01.990Z\",\r\n \"ReleaseName\": \"ReportPdfGen\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-14\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022390,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"c5d6e7f8-a9b0-4193-bc2d-3e4f50617283\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-17T09:40:06.700Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d6e7f8a9-b0c1-4204-cd3e-4f5061728394\",\r\n \"Id\": 682459887\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d6e7f8a9-b0c1-4204-cd3e-4f5061728394", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T09:40:02.270Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T09:40:03.980Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-17T09:40:05.310Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\Report.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-17T09:40:06.630Z\",\r\n \"Message\": \"Export to PDF: The export file path must specify a .pdf file. Value was 'C:\\\\Reports\\\\MonthlyReport'.\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Export to PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-17T09:40:06.680Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5d6e7f8-a9b0-4193-bc2d-3e4f50617283\",\r\n \"Name\": \"Report PDFs\",\r\n \"Path\": \"Report PDFs\",\r\n \"Description\": \"Production folder for report PDF generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/task.yaml index 3bd62a4ade..12c72a7c2f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-output-path-format/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/README.md new file mode 100644 index 0000000000..8d5c41a3ac --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/README.md @@ -0,0 +1,44 @@ +# Save Document as PDF — COM Wrong-Thread (External Word Attached + Closed Mid-Run) + +This scenario reproduces a `Save Document as PDF` (`WordExportToPdf`) COM +failure that faults casting the document to +`Microsoft.Office.Interop.Word._Document` with +`0x8001010E RPC_E_WRONG_THREAD`. A Microsoft Word (`WINWORD.EXE`) was already +open on the host when the run started; `Word Application Scope` attached to +that external instance (it exposes no isolated-instance option), and the user +closed that Word window mid-run before the export. + +## What this scenario uncovers + +**Root Cause:** `Word Application Scope` reused an already-running external +`WINWORD.EXE`, so the document's `_Document` COM object lived on that external +process's STA apartment. When the user closed that Word window mid-run, the +apartment owning the proxy was torn down / replaced, and the export's +cross-apartment `QueryInterface` failed with `RPC_E_WRONG_THREAD` (the +signature of a *replaced* apartment — distinct from `0x80010108 +RPC_E_DISCONNECTED`, a clean server death). The workflow is structurally +correct: the export is the sole child of the scope that opened the document. + +This maps to: +`references/activity-packages/word-activities/playbooks/word-export-pdf-com-wrong-thread.md` + +The correct agent behavior is to tie `RPC_E_WRONG_THREAD` + the IID +`{0002096B-...}` (`_Document`) to the external-attach / mid-run-close cause +(using the attach + close warnings in the log), and recommend ensuring no +external Word is open during the run (let the scope own its own instance), +with an A/B re-run as confirmation — without blaming the document or the +output path. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored `ContractPdfExport`; `Word Application Scope` opens `data\Contract.docx`, `Save Document as PDF` → `data\out\Contract.pdf` (structurally correct, export is the sole child) | + +> **Note on fixtures.** Synthetic. The attended job ran on `MOCK-HOST` while +> an external Word was open; the log's attach + close warnings are the +> decisive evidence, not the workflow structure. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/364c146301.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/364c146301.json deleted file mode 100644 index e8a90df674..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/364c146301.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182", - "StartTime": "2026-06-19T14:22:11.210Z", - "EndTime": "2026-06-19T14:22:18.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'.\r\n\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).", - "CreationTime": "2026-06-19T14:22:10.980Z", - "ReleaseName": "ContractPdfExport", - "Type": "Attended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Attended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6033600, - "OrganizationUnitId": 7791618, - "FolderKey": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "MOCK-HOST\\analyst", - "OrchestratorUserIdentity": "analyst", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-19T14:22:18.640Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidCastException", - "ActivityName": "WordExportToPdf_1", - "ActivityDisplayName": "Save Document as PDF", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182", - "Id": 682471101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/3c9f588665.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/3c9f588665.json deleted file mode 100644 index 4c7a07af64..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/3c9f588665.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182", - "State": "Faulted", - "ProcessName": "ContractPdfExport", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-19T14:22:11.210Z", - "EndTime": "2026-06-19T14:22:18.640Z", - "CreationTime": "2026-06-19T14:22:10.980Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/README.md deleted file mode 100644 index 26b0c51455..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Save Document as PDF — COM Wrong-Thread (External Word Attached + Closed Mid-Run) - -This scenario reproduces a `Save Document as PDF` (`WordExportToPdf`) COM -failure that faults casting the document to -`Microsoft.Office.Interop.Word._Document` with -`0x8001010E RPC_E_WRONG_THREAD`. A Microsoft Word (`WINWORD.EXE`) was already -open on the host when the run started; `Word Application Scope` attached to -that external instance (it exposes no isolated-instance option), and the user -closed that Word window mid-run before the export. - -## What this scenario uncovers - -**Root Cause:** `Word Application Scope` reused an already-running external -`WINWORD.EXE`, so the document's `_Document` COM object lived on that external -process's STA apartment. When the user closed that Word window mid-run, the -apartment owning the proxy was torn down / replaced, and the export's -cross-apartment `QueryInterface` failed with `RPC_E_WRONG_THREAD` (the -signature of a *replaced* apartment — distinct from `0x80010108 -RPC_E_DISCONNECTED`, a clean server death). The workflow is structurally -correct: the export is the sole child of the scope that opened the document. - -This maps to: -`references/activity-packages/word-activities/playbooks/word-export-pdf-com-wrong-thread.md` - -The correct agent behavior is to tie `RPC_E_WRONG_THREAD` + the IID -`{0002096B-...}` (`_Document`) to the external-attach / mid-run-close cause -(using the attach + close warnings in the log), and recommend ensuring no -external Word is open during the run (let the scope own its own instance), -with an A/B re-run as confirmation — without blaming the document or the -output path. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored `ContractPdfExport`; `Word Application Scope` opens `data\Contract.docx`, `Save Document as PDF` → `data\out\Contract.pdf` (structurally correct, export is the sole child) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `or jobs get` Info and the logs show the wrong-thread cast, with attach + mid-run-close warnings | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Synthetic. The attended job ran on `MOCK-HOST` while -> an external Word was open; the log's attach + close warnings are the -> decisive evidence, not the workflow structure. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/ab8129f279.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/ab8129f279.json deleted file mode 100644 index d3e650d7b0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/ab8129f279.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-19T14:22:11.260Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T14:22:12.520Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-19T14:22:13.040Z", - "Message": "[Word Application Scope] A Microsoft Word instance was already running on the host; the scope attached to the existing WINWORD.EXE (no isolated-instance option is available)." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-19T14:22:14.180Z", - "Message": "[Word Application Scope] Document 'data\\Contract.docx' opened in the attached Word instance." - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-19T14:22:16.700Z", - "Message": "[Word Application Scope] The attached Microsoft Word window was closed by the user before the scope completed." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-19T14:22:18.560Z", - "Message": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T14:22:18.620Z", - "Message": "Execution ended in: 00:00:07" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/bfd840ba38.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/bfd840ba38.json deleted file mode 100644 index de20cfb093..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/bfd840ba38.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-19T14:22:18.560Z", - "Message": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/df2860582c.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/df2860582c.json deleted file mode 100644 index 8d8ed33112..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/df2860582c.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071", - "Name": "Contracts", - "Path": "Contracts", - "Description": "Production folder for contract document processing.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/manifest.json deleted file mode 100644 index 6c486e4e08..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the export-pdf wrongthread-external-close scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "df2860582c.json" - }, - { - "match": "or jobs list --folder-key \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071\" --state Faulted --output json", - "file": "3c9f588665.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071 --state Faulted --output json", - "file": "3c9f588665.json" - }, - { - "match": "or jobs get \"b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182\" --output json", - "file": "364c146301.json" - }, - { - "match": "or jobs get b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182 --output json", - "file": "364c146301.json" - }, - { - "match": "or jobs logs \"b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182\" --level Error --output json", - "file": "bfd840ba38.json" - }, - { - "match": "or jobs logs b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182 --level Error --output json", - "file": "bfd840ba38.json" - }, - { - "match": "or jobs logs \"b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182\" --output json", - "file": "ab8129f279.json" - }, - { - "match": "or jobs logs b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182 --output json", - "file": "ab8129f279.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/uip-fixture.json new file mode 100644 index 0000000000..c67efd88d1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ContractPdfExport\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-19T14:22:11.210Z\",\r\n \"EndTime\": \"2026-06-19T14:22:18.640Z\",\r\n \"CreationTime\": \"2026-06-19T14:22:10.980Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T14:22:18.560Z\",\r\n \"Message\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182\",\r\n \"StartTime\": \"2026-06-19T14:22:11.210Z\",\r\n \"EndTime\": \"2026-06-19T14:22:18.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'.\\r\\n\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\",\r\n \"CreationTime\": \"2026-06-19T14:22:10.980Z\",\r\n \"ReleaseName\": \"ContractPdfExport\",\r\n \"Type\": \"Attended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Attended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6033600,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"MOCK-HOST\\\\analyst\",\r\n \"OrchestratorUserIdentity\": \"analyst\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-19T14:22:18.640Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidCastException\",\r\n \"ActivityName\": \"WordExportToPdf_1\",\r\n \"ActivityDisplayName\": \"Save Document as PDF\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182\",\r\n \"Id\": 682471101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b2c3d4e5-f6a7-4819-8b2c-3d4e5f607182", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T14:22:11.260Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T14:22:12.520Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-19T14:22:13.040Z\",\r\n \"Message\": \"[Word Application Scope] A Microsoft Word instance was already running on the host; the scope attached to the existing WINWORD.EXE (no isolated-instance option is available).\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-19T14:22:14.180Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\Contract.docx' opened in the attached Word instance.\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-19T14:22:16.700Z\",\r\n \"Message\": \"[Word Application Scope] The attached Microsoft Word window was closed by the user before the scope completed.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T14:22:18.560Z\",\r\n \"Message\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T14:22:18.620Z\",\r\n \"Message\": \"Execution ended in: 00:00:07\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4708-9a1b-2c3d4e5f6071\",\r\n \"Name\": \"Contracts\",\r\n \"Path\": \"Contracts\",\r\n \"Description\": \"Production folder for contract document processing.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/task.yaml index b50a4e3ef5..9d6dab52e6 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-external-close/task.yaml @@ -27,22 +27,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/README.md new file mode 100644 index 0000000000..23ab628e0b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/README.md @@ -0,0 +1,44 @@ +# Save Document as PDF — COM Wrong-Thread (Non-Creator Thread / Parallel) + +This scenario reproduces a `Save Document as PDF` (`WordExportToPdf`) COM +failure that faults casting the document to +`Microsoft.Office.Interop.Word._Document` with +`0x8001010E RPC_E_WRONG_THREAD`. A **`Parallel`** activity between +scope-open and the export runs the export on a worker thread that is **not** +the STA that created the document. + +## What this scenario uncovers + +**Root Cause:** The `_Document` COM proxy is valid only on the apartment/thread +that created it (the thread `Word Application Scope` opened the document on). +The workflow nests `Save Document as PDF` inside a `Parallel` activity, so the +export executes on a different (non-creator) thread; its cross-apartment +`QueryInterface` for `_Document` fails with `RPC_E_WRONG_THREAD`. The run is +attended/foreground with no external Word — the cause is the threading +construct in the workflow source, visible both in `Main.xaml` and in the +stack trace (`at Parallel "Parallel — export + audit log"`). + +This maps to: +`references/activity-packages/word-activities/playbooks/word-export-pdf-com-wrong-thread.md` + +The correct agent behavior is to tie `RPC_E_WRONG_THREAD` + the IID +`{0002096B-...}` (`_Document`) to the `Parallel`/non-creator-thread cause +(reading the `Parallel` in `Main.xaml` and/or the stack frame), and recommend +moving the export onto the same thread/scope that opened the document (remove +the `Parallel` around the export) — without blaming the document or the +output path. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored `BulkDocPdf`; `Word Application Scope` opens `data\Invoice.docx`, with a `Parallel` inside the scope body whose branch runs `Save Document as PDF` → `data\out\Invoice.pdf` | + +> **Note on fixtures.** Synthetic. The decisive evidence is the `Parallel` +> wrapping the export in `Main.xaml` (mirrored by the `Parallel` frame in the +> stack trace), on an attended run with no external Word — isolating the +> non-creator-thread cause. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/2f9b931af0.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/2f9b931af0.json deleted file mode 100644 index 3f0290da64..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/2f9b931af0.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f6a7b8c9-d0e1-4c4d-8f60-718293041526", - "State": "Faulted", - "ProcessName": "BulkDocPdf", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-19T10:05:21.330Z", - "EndTime": "2026-06-19T10:05:27.910Z", - "CreationTime": "2026-06-19T10:05:21.100Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/764af80621.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/764af80621.json deleted file mode 100644 index 97ba702c7c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/764af80621.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f6a7b8c9-d0e1-4c4d-8f60-718293041526", - "StartTime": "2026-06-19T10:05:21.330Z", - "EndTime": "2026-06-19T10:05:27.910Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'.\r\n\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at Parallel \"Parallel — export + audit log\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).", - "CreationTime": "2026-06-19T10:05:21.100Z", - "ReleaseName": "BulkDocPdf", - "Type": "Attended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Attended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6055800, - "OrganizationUnitId": 7791618, - "FolderKey": "e5f6a7b8-c9d0-4b3c-9e5f-607182930415", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "MOCK-HOST\\operator", - "OrchestratorUserIdentity": "operator", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-19T10:05:27.910Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidCastException", - "ActivityName": "WordExportToPdf_1", - "ActivityDisplayName": "Save Document as PDF", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "f6a7b8c9-d0e1-4c4d-8f60-718293041526", - "Id": 682493307 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/README.md deleted file mode 100644 index 5df389c5b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Save Document as PDF — COM Wrong-Thread (Non-Creator Thread / Parallel) - -This scenario reproduces a `Save Document as PDF` (`WordExportToPdf`) COM -failure that faults casting the document to -`Microsoft.Office.Interop.Word._Document` with -`0x8001010E RPC_E_WRONG_THREAD`. A **`Parallel`** activity between -scope-open and the export runs the export on a worker thread that is **not** -the STA that created the document. - -## What this scenario uncovers - -**Root Cause:** The `_Document` COM proxy is valid only on the apartment/thread -that created it (the thread `Word Application Scope` opened the document on). -The workflow nests `Save Document as PDF` inside a `Parallel` activity, so the -export executes on a different (non-creator) thread; its cross-apartment -`QueryInterface` for `_Document` fails with `RPC_E_WRONG_THREAD`. The run is -attended/foreground with no external Word — the cause is the threading -construct in the workflow source, visible both in `Main.xaml` and in the -stack trace (`at Parallel "Parallel — export + audit log"`). - -This maps to: -`references/activity-packages/word-activities/playbooks/word-export-pdf-com-wrong-thread.md` - -The correct agent behavior is to tie `RPC_E_WRONG_THREAD` + the IID -`{0002096B-...}` (`_Document`) to the `Parallel`/non-creator-thread cause -(reading the `Parallel` in `Main.xaml` and/or the stack frame), and recommend -moving the export onto the same thread/scope that opened the document (remove -the `Parallel` around the export) — without blaming the document or the -output path. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored `BulkDocPdf`; `Word Application Scope` opens `data\Invoice.docx`, with a `Parallel` inside the scope body whose branch runs `Save Document as PDF` → `data\out\Invoice.pdf` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; `or jobs get` Info and the logs show the wrong-thread cast with a `Parallel` frame in the stack | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Synthetic. The decisive evidence is the `Parallel` -> wrapping the export in `Main.xaml` (mirrored by the `Parallel` frame in the -> stack trace), on an attended run with no external Word — isolating the -> non-creator-thread cause. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/ae054bd23c.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/ae054bd23c.json deleted file mode 100644 index 6733fa0434..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/ae054bd23c.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e5f6a7b8-c9d0-4b3c-9e5f-607182930415", - "Name": "DocOps", - "Path": "DocOps", - "Description": "Bulk document operations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/e9ce5dc5e4.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/e9ce5dc5e4.json deleted file mode 100644 index d242ed73e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/e9ce5dc5e4.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-19T10:05:21.380Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T10:05:22.640Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-19T10:05:24.010Z", - "Message": "[Word Application Scope] Document 'data\\Invoice.docx' opened" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T10:05:24.520Z", - "Message": "Dispatching invoice export to PDF" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-19T10:05:27.840Z", - "Message": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at Parallel \"Parallel — export + audit log\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T10:05:27.890Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/f29887a6a5.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/f29887a6a5.json deleted file mode 100644 index 8bde2727ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/f29887a6a5.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-19T10:05:27.840Z", - "Message": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at Parallel \"Parallel — export + audit log\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/manifest.json deleted file mode 100644 index db8baedcfd..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the export-pdf wrongthread-parallel scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "ae054bd23c.json" - }, - { - "match": "or jobs list --folder-key \"e5f6a7b8-c9d0-4b3c-9e5f-607182930415\" --state Faulted --output json", - "file": "2f9b931af0.json" - }, - { - "match": "or jobs list --folder-key e5f6a7b8-c9d0-4b3c-9e5f-607182930415 --state Faulted --output json", - "file": "2f9b931af0.json" - }, - { - "match": "or jobs get \"f6a7b8c9-d0e1-4c4d-8f60-718293041526\" --output json", - "file": "764af80621.json" - }, - { - "match": "or jobs get f6a7b8c9-d0e1-4c4d-8f60-718293041526 --output json", - "file": "764af80621.json" - }, - { - "match": "or jobs logs \"f6a7b8c9-d0e1-4c4d-8f60-718293041526\" --level Error --output json", - "file": "f29887a6a5.json" - }, - { - "match": "or jobs logs f6a7b8c9-d0e1-4c4d-8f60-718293041526 --level Error --output json", - "file": "f29887a6a5.json" - }, - { - "match": "or jobs logs \"f6a7b8c9-d0e1-4c4d-8f60-718293041526\" --output json", - "file": "e9ce5dc5e4.json" - }, - { - "match": "or jobs logs f6a7b8c9-d0e1-4c4d-8f60-718293041526 --output json", - "file": "e9ce5dc5e4.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/uip-fixture.json new file mode 100644 index 0000000000..d992577942 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e5f6a7b8-c9d0-4b3c-9e5f-607182930415", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f6a7b8c9-d0e1-4c4d-8f60-718293041526\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BulkDocPdf\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-19T10:05:21.330Z\",\r\n \"EndTime\": \"2026-06-19T10:05:27.910Z\",\r\n \"CreationTime\": \"2026-06-19T10:05:21.100Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f6a7b8c9-d0e1-4c4d-8f60-718293041526", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T10:05:27.840Z\",\r\n \"Message\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at Parallel \\\"Parallel — export + audit log\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f6a7b8c9-d0e1-4c4d-8f60-718293041526", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f6a7b8c9-d0e1-4c4d-8f60-718293041526\",\r\n \"StartTime\": \"2026-06-19T10:05:21.330Z\",\r\n \"EndTime\": \"2026-06-19T10:05:27.910Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'.\\r\\n\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at Parallel \\\"Parallel — export + audit log\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\",\r\n \"CreationTime\": \"2026-06-19T10:05:21.100Z\",\r\n \"ReleaseName\": \"BulkDocPdf\",\r\n \"Type\": \"Attended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Attended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6055800,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"e5f6a7b8-c9d0-4b3c-9e5f-607182930415\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"MOCK-HOST\\\\operator\",\r\n \"OrchestratorUserIdentity\": \"operator\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-19T10:05:27.910Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidCastException\",\r\n \"ActivityName\": \"WordExportToPdf_1\",\r\n \"ActivityDisplayName\": \"Save Document as PDF\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"f6a7b8c9-d0e1-4c4d-8f60-718293041526\",\r\n \"Id\": 682493307\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f6a7b8c9-d0e1-4c4d-8f60-718293041526", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T10:05:21.380Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T10:05:22.640Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-19T10:05:24.010Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\Invoice.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T10:05:24.520Z\",\r\n \"Message\": \"Dispatching invoice export to PDF\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T10:05:27.840Z\",\r\n \"Message\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at Parallel \\\"Parallel — export + audit log\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T10:05:27.890Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e5f6a7b8-c9d0-4b3c-9e5f-607182930415\",\r\n \"Name\": \"DocOps\",\r\n \"Path\": \"DocOps\",\r\n \"Description\": \"Bulk document operations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/task.yaml index 76f5b08802..6c73b6d2a1 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-parallel/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/README.md new file mode 100644 index 0000000000..ac3560d0c4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/README.md @@ -0,0 +1,43 @@ +# Save Document as PDF — COM Wrong-Thread (Unattended Session-0 / Off-STA) + +This scenario reproduces a `Save Document as PDF` (`WordExportToPdf`) COM +failure that faults casting the document to +`Microsoft.Office.Interop.Word._Document` with +`0x8001010E RPC_E_WRONG_THREAD` on an **unattended Session-0 / background** +runtime. No external Word is involved — the run surface itself is the cause. + +## What this scenario uncovers + +**Root Cause:** Word COM interop requires an interactive STA session. The job +ran **unattended in Session 0** (non-interactive), so the worker thread that +executed `Save Document as PDF` is not the interactive STA that can own the +`_Document` COM object, and the cross-apartment `QueryInterface` failed with +`RPC_E_WRONG_THREAD`. The workflow is structurally correct (the export is the +sole child of the scope that opened the document) and there is **no** external +Word warning in the log — distinguishing this from the external-attach / +mid-run-close cause. + +This maps to: +`references/activity-packages/word-activities/playbooks/word-export-pdf-com-wrong-thread.md` + +The correct agent behavior is to tie `RPC_E_WRONG_THREAD` + the IID +`{0002096B-...}` (`_Document`) to the off-STA / Session-0 run surface (using +the unattended/non-interactive run evidence and the absence of any external +Word), and recommend either running the automation attended in an interactive +session or migrating the export to the **System Word** (background) +activities — without blaming the document or the output path. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored `NightlyReportPdf`; `Word Application Scope` opens `data\Report.docx`, `Save Document as PDF` → `data\out\Report.pdf` (structurally correct); `project.json` is a background process (`requiresUserInteraction: false`) | + +> **Note on fixtures.** Synthetic. The decisive evidence is the unattended / +> Session-0 / non-interactive run surface plus the absence of any external +> Word warning — the cause is the run surface, not an attached Word or the +> workflow structure. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/17b2e56e28.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/17b2e56e28.json deleted file mode 100644 index bd3cd09c83..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/17b2e56e28.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-19T02:00:04.160Z", - "Message": "Execution started (Unattended, Session 0, non-interactive)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T02:00:05.420Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-19T02:00:07.180Z", - "Message": "[Word Application Scope] Document 'data\\Report.docx' opened" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-19T02:00:10.470Z", - "Message": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-19T02:00:10.510Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/27883962a2.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/27883962a2.json deleted file mode 100644 index 37980290c7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/27883962a2.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304", - "State": "Faulted", - "ProcessName": "NightlyReportPdf", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-19T02:00:04.110Z", - "EndTime": "2026-06-19T02:00:10.530Z", - "CreationTime": "2026-06-19T02:00:03.900Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/5b2c6d0196.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/5b2c6d0196.json deleted file mode 100644 index 26f1125bf5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/5b2c6d0196.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c3d4e5f6-a7b8-491a-9c3d-4e5f60718293", - "Name": "Reports", - "Path": "Reports", - "Description": "Unattended nightly report generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/README.md deleted file mode 100644 index 9f4fcb68b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/README.md +++ /dev/null @@ -1,42 +0,0 @@ -# Save Document as PDF — COM Wrong-Thread (Unattended Session-0 / Off-STA) - -This scenario reproduces a `Save Document as PDF` (`WordExportToPdf`) COM -failure that faults casting the document to -`Microsoft.Office.Interop.Word._Document` with -`0x8001010E RPC_E_WRONG_THREAD` on an **unattended Session-0 / background** -runtime. No external Word is involved — the run surface itself is the cause. - -## What this scenario uncovers - -**Root Cause:** Word COM interop requires an interactive STA session. The job -ran **unattended in Session 0** (non-interactive), so the worker thread that -executed `Save Document as PDF` is not the interactive STA that can own the -`_Document` COM object, and the cross-apartment `QueryInterface` failed with -`RPC_E_WRONG_THREAD`. The workflow is structurally correct (the export is the -sole child of the scope that opened the document) and there is **no** external -Word warning in the log — distinguishing this from the external-attach / -mid-run-close cause. - -This maps to: -`references/activity-packages/word-activities/playbooks/word-export-pdf-com-wrong-thread.md` - -The correct agent behavior is to tie `RPC_E_WRONG_THREAD` + the IID -`{0002096B-...}` (`_Document`) to the off-STA / Session-0 run surface (using -the unattended/non-interactive run evidence and the absence of any external -Word), and recommend either running the automation attended in an interactive -session or migrating the export to the **System Word** (background) -activities — without blaming the document or the output path. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored `NightlyReportPdf`; `Word Application Scope` opens `data\Report.docx`, `Save Document as PDF` → `data\out\Report.pdf` (structurally correct); `project.json` is a background process (`requiresUserInteraction: false`) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses; the job is `Unattended` / Schedule-triggered, the log notes Session 0 / non-interactive, and the export faults with the wrong-thread cast | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Synthetic. The decisive evidence is the unattended / -> Session-0 / non-interactive run surface plus the absence of any external -> Word warning — the cause is the run surface, not an attached Word or the -> workflow structure. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/e48085787d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/e48085787d.json deleted file mode 100644 index d7475b610f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/e48085787d.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-19T02:00:10.470Z", - "Message": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/fc8774f2dc.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/fc8774f2dc.json deleted file mode 100644 index 24886c5a88..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/fc8774f2dc.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304", - "StartTime": "2026-06-19T02:00:04.110Z", - "EndTime": "2026-06-19T02:00:10.530Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "d5e6f7a8-b9c0-4b3c-8d5e-6f7182930415", - "Info": "Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'.\r\n\r\nin Main.xaml\r\n at WordExportToPdf \"Save Document as PDF\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).", - "CreationTime": "2026-06-19T02:00:03.900Z", - "ReleaseName": "NightlyReportPdf", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 6044700, - "OrganizationUnitId": 7791618, - "FolderKey": "c3d4e5f6-a7b8-491a-9c3d-4e5f60718293", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-19T02:00:10.530Z", - "JobError": { - "ErrorCode": "Activity", - "Type": "InvalidCastException", - "ActivityName": "WordExportToPdf_1", - "ActivityDisplayName": "Save Document as PDF", - "WorkflowFilePath": "Main.xaml" - }, - "ErrorCode": "Activity", - "TraceId": "d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304", - "Id": 682482204 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/manifest.json deleted file mode 100644 index ac6179acb4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the export-pdf wrongthread-session0 scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "5b2c6d0196.json" - }, - { - "match": "or jobs list --folder-key \"c3d4e5f6-a7b8-491a-9c3d-4e5f60718293\" --state Faulted --output json", - "file": "27883962a2.json" - }, - { - "match": "or jobs list --folder-key c3d4e5f6-a7b8-491a-9c3d-4e5f60718293 --state Faulted --output json", - "file": "27883962a2.json" - }, - { - "match": "or jobs get \"d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304\" --output json", - "file": "fc8774f2dc.json" - }, - { - "match": "or jobs get d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304 --output json", - "file": "fc8774f2dc.json" - }, - { - "match": "or jobs logs \"d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304\" --level Error --output json", - "file": "e48085787d.json" - }, - { - "match": "or jobs logs d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304 --level Error --output json", - "file": "e48085787d.json" - }, - { - "match": "or jobs logs \"d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304\" --output json", - "file": "17b2e56e28.json" - }, - { - "match": "or jobs logs d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304 --output json", - "file": "17b2e56e28.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/uip-fixture.json new file mode 100644 index 0000000000..e1223d0246 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c3d4e5f6-a7b8-491a-9c3d-4e5f60718293", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReportPdf\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-19T02:00:04.110Z\",\r\n \"EndTime\": \"2026-06-19T02:00:10.530Z\",\r\n \"CreationTime\": \"2026-06-19T02:00:03.900Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T02:00:10.470Z\",\r\n \"Message\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304\",\r\n \"StartTime\": \"2026-06-19T02:00:04.110Z\",\r\n \"EndTime\": \"2026-06-19T02:00:10.530Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"d5e6f7a8-b9c0-4b3c-8d5e-6f7182930415\",\r\n \"Info\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'.\\r\\n\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\",\r\n \"CreationTime\": \"2026-06-19T02:00:03.900Z\",\r\n \"ReleaseName\": \"NightlyReportPdf\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 6044700,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"c3d4e5f6-a7b8-491a-9c3d-4e5f60718293\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-19T02:00:10.530Z\",\r\n \"JobError\": {\r\n \"ErrorCode\": \"Activity\",\r\n \"Type\": \"InvalidCastException\",\r\n \"ActivityName\": \"WordExportToPdf_1\",\r\n \"ActivityDisplayName\": \"Save Document as PDF\",\r\n \"WorkflowFilePath\": \"Main.xaml\"\r\n },\r\n \"ErrorCode\": \"Activity\",\r\n \"TraceId\": \"d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304\",\r\n \"Id\": 682482204\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d4e5f6a7-b8c9-4a2b-8d4e-5f6071829304", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T02:00:04.160Z\",\r\n \"Message\": \"Execution started (Unattended, Session 0, non-interactive)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T02:00:05.420Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-19T02:00:07.180Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\Report.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T02:00:10.470Z\",\r\n \"Message\": \"Save Document as PDF: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Office.Interop.Word._Document'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{0002096B-0000-0000-C000-000000000046}' failed due to the following error: The application called an interface that was marshalled for a different thread. (0x8001010E (RPC_E_WRONG_THREAD)).\\r\\nin Main.xaml\\r\\n at WordExportToPdf \\\"Save Document as PDF\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-19T02:00:10.510Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-491a-9c3d-4e5f60718293\",\r\n \"Name\": \"Reports\",\r\n \"Path\": \"Reports\",\r\n \"Description\": \"Unattended nightly report generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/task.yaml index 509b9a3594..3505f3efe5 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-export-pdf-wrongthread-session0/task.yaml @@ -26,22 +26,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/README.md new file mode 100644 index 0000000000..bc3f9c9390 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/README.md @@ -0,0 +1,45 @@ +# Read Text Failure - Standalone System Activity Fails on Legacy .doc + +This scenario reproduces a `Read Text` failure where the standalone +`System > File > Word Document` `Read Text` activity (OpenXML, `.docx`-only) +faults on a legacy binary `.doc` file with a `FileFormatException`. + +## What this scenario uncovers + +**Root Cause:** The standalone System Read Text parses via OpenXML, which +understands `.docx` only. The input `Contract.doc` is a legacy binary `.doc`, +so it throws `FileFormatException` ("not a valid Office Open XML document"). +Newer `.docx` files work; older `.doc` files fail. The file exists and the +path is correct — it's the format the OpenXML reader can't parse. + +This maps to: +`references/activity-packages/word-activities/playbooks/read-text-doc-format.md` + +The correct agent behavior is to tie the fault to the `.doc`-vs-`.docx` +limitation of the standalone activity and recommend reading through a +`Use Word File` (Interop opens both) or converting to `.docx` first — not +blaming a missing/corrupt file, the path, or the Word install. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; standalone `Read Text` with its own `FileName="data\Contract.doc"`, project depends only on `UiPath.System.Activities` | + +> **Note on fixtures.** The faulted job's `FileFormatException` explicitly +> says only `.docx` is supported and names `Contract.doc`; the project's +> dependency on `UiPath.System.Activities` only (no `UiPath.Word.Activities`) +> confirms the standalone OpenXML surface with no Interop path. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `read-text-doc-format.md` +- Agent identified the standalone System Read Text being `.docx`-only failing + on a legacy `.doc` as the cause and recommended reading via `Use Word File` + (Interop) or converting to `.docx` first — without blaming a + missing/corrupt file, the path, or the install + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/3287bd3782.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/3287bd3782.json deleted file mode 100644 index b0a847e6d0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/3287bd3782.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T09:14:04.190Z", - "Message": "The file 'C:\\UiPath\\Legacy Docs\\data\\Contract.doc' is not a valid Office Open XML document. Only .docx files are supported by this activity.\r\nin Main.xaml\r\n at WordReadTextStandalone \"Read Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/73fe655cfd.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/73fe655cfd.json deleted file mode 100644 index c4df56c9da..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/73fe655cfd.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7", - "Name": "Legacy Docs", - "Path": "Legacy Docs", - "Description": "Production folder for legacy document processing.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/79e2ec2aac.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/79e2ec2aac.json deleted file mode 100644 index 45b097c0b3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/79e2ec2aac.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8", - "State": "Faulted", - "ProcessName": "LegacyDocReader", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-11", - "MachineKey": "", - "StartTime": "2026-06-16T09:14:02.110Z", - "EndTime": "2026-06-16T09:14:04.260Z", - "CreationTime": "2026-06-16T09:14:01.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/README.md deleted file mode 100644 index 733e164c44..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/README.md +++ /dev/null @@ -1,44 +0,0 @@ -# Read Text Failure - Standalone System Activity Fails on Legacy .doc - -This scenario reproduces a `Read Text` failure where the standalone -`System > File > Word Document` `Read Text` activity (OpenXML, `.docx`-only) -faults on a legacy binary `.doc` file with a `FileFormatException`. - -## What this scenario uncovers - -**Root Cause:** The standalone System Read Text parses via OpenXML, which -understands `.docx` only. The input `Contract.doc` is a legacy binary `.doc`, -so it throws `FileFormatException` ("not a valid Office Open XML document"). -Newer `.docx` files work; older `.doc` files fail. The file exists and the -path is correct — it's the format the OpenXML reader can't parse. - -This maps to: -`references/activity-packages/word-activities/playbooks/read-text-doc-format.md` - -The correct agent behavior is to tie the fault to the `.doc`-vs-`.docx` -limitation of the standalone activity and recommend reading through a -`Use Word File` (Interop opens both) or converting to `.docx` first — not -blaming a missing/corrupt file, the path, or the Word install. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; standalone `Read Text` with its own `FileName="data\Contract.doc"`, project depends only on `UiPath.System.Activities` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's `FileFormatException` explicitly -> says only `.docx` is supported and names `Contract.doc`; the project's -> dependency on `UiPath.System.Activities` only (no `UiPath.Word.Activities`) -> confirms the standalone OpenXML surface with no Interop path. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `read-text-doc-format.md` -- Agent identified the standalone System Read Text being `.docx`-only failing - on a legacy `.doc` as the cause and recommended reading via `Use Word File` - (Interop) or converting to `.docx` first — without blaming a - missing/corrupt file, the path, or the install diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/b5a2c3c843.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/b5a2c3c843.json deleted file mode 100644 index f6fc562a0a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/b5a2c3c843.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8", - "StartTime": "2026-06-16T09:14:02.110Z", - "EndTime": "2026-06-16T09:14:04.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "b9c0d1e2-f3a4-4315-bf50-14a5b6c7d8e9", - "Info": "Read Text: The file 'C:\\UiPath\\Legacy Docs\\data\\Contract.doc' is not a valid Office Open XML document. Only .docx files are supported by this activity.\r\n\r\nin Main.xaml\r\n at WordReadTextStandalone \"Read Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileFormatException: File contains corrupted data. (Expected an Office Open XML package; the file may be a legacy .doc binary.)", - "CreationTime": "2026-06-16T09:14:01.880Z", - "ReleaseName": "LegacyDocReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-11", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022060, - "OrganizationUnitId": 7791618, - "FolderKey": "a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T09:14:04.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8", - "Id": 682456554 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/c5f8ba0012.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/c5f8ba0012.json deleted file mode 100644 index f91397b5e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/c5f8ba0012.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T09:14:02.160Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T09:14:03.420Z", - "Message": "[Read Text] Opening 'data\\Contract.doc'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T09:14:04.190Z", - "Message": "The file 'C:\\UiPath\\Legacy Docs\\data\\Contract.doc' is not a valid Office Open XML document. Only .docx files are supported by this activity.\r\nin Main.xaml\r\n at WordReadTextStandalone \"Read Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T09:14:04.230Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/manifest.json deleted file mode 100644 index 79cc2ddb59..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the read-text doc-format scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "73fe655cfd.json" - }, - { - "match": "or jobs list --folder-key \"a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7\" --state Faulted --output json", - "file": "79e2ec2aac.json" - }, - { - "match": "or jobs list --folder-key a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7 --state Faulted --output json", - "file": "79e2ec2aac.json" - }, - { - "match": "or jobs get \"b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8\" --output json", - "file": "b5a2c3c843.json" - }, - { - "match": "or jobs get b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8 --output json", - "file": "b5a2c3c843.json" - }, - { - "match": "or jobs logs \"b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8\" --level Error --output json", - "file": "3287bd3782.json" - }, - { - "match": "or jobs logs b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8 --level Error --output json", - "file": "3287bd3782.json" - }, - { - "match": "or jobs logs \"b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8\" --output json", - "file": "c5f8ba0012.json" - }, - { - "match": "or jobs logs b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8 --output json", - "file": "c5f8ba0012.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/uip-fixture.json new file mode 100644 index 0000000000..0c829f25e2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LegacyDocReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-11\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T09:14:02.110Z\",\r\n \"EndTime\": \"2026-06-16T09:14:04.260Z\",\r\n \"CreationTime\": \"2026-06-16T09:14:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T09:14:04.190Z\",\r\n \"Message\": \"The file 'C:\\\\UiPath\\\\Legacy Docs\\\\data\\\\Contract.doc' is not a valid Office Open XML document. Only .docx files are supported by this activity.\\r\\nin Main.xaml\\r\\n at WordReadTextStandalone \\\"Read Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8\",\r\n \"StartTime\": \"2026-06-16T09:14:02.110Z\",\r\n \"EndTime\": \"2026-06-16T09:14:04.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"b9c0d1e2-f3a4-4315-bf50-14a5b6c7d8e9\",\r\n \"Info\": \"Read Text: The file 'C:\\\\UiPath\\\\Legacy Docs\\\\data\\\\Contract.doc' is not a valid Office Open XML document. Only .docx files are supported by this activity.\\r\\n\\r\\nin Main.xaml\\r\\n at WordReadTextStandalone \\\"Read Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileFormatException: File contains corrupted data. (Expected an Office Open XML package; the file may be a legacy .doc binary.)\",\r\n \"CreationTime\": \"2026-06-16T09:14:01.880Z\",\r\n \"ReleaseName\": \"LegacyDocReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-11\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022060,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T09:14:04.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8\",\r\n \"Id\": 682456554\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b8c9d0e1-f2a3-4204-ae4f-0314a5b6c7d8", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T09:14:02.160Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T09:14:03.420Z\",\r\n \"Message\": \"[Read Text] Opening 'data\\\\Contract.doc'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T09:14:04.190Z\",\r\n \"Message\": \"The file 'C:\\\\UiPath\\\\Legacy Docs\\\\data\\\\Contract.doc' is not a valid Office Open XML document. Only .docx files are supported by this activity.\\r\\nin Main.xaml\\r\\n at WordReadTextStandalone \\\"Read Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T09:14:04.230Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a7b8c9d0-e1f2-4193-9d3e-920314a5b6c7\",\r\n \"Name\": \"Legacy Docs\",\r\n \"Path\": \"Legacy Docs\",\r\n \"Description\": \"Production folder for legacy document processing.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/task.yaml index 203bc2904f..b8350a6eba 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-doc-format/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/README.md new file mode 100644 index 0000000000..b7f96d6fd1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/README.md @@ -0,0 +1,47 @@ +# Read Text Failure - Activity Outside Its Container + +This scenario reproduces a `Read Text` failure where the modern Word-pack +`Read Text` activity is dropped **outside** a `Use Word File` / +`Word Application Scope` and faults as invalid — it has no file input of its +own and reads the container's open document. + +## What this scenario uncovers + +**Root Cause:** The Word-pack `Read Text` sits loose in the sequence with no +container. It reads the document held open by a surrounding scope; outside +one it is invalid, so the job faults immediately with "The 'Read Text' +activity must be placed inside a 'Use Word File' or 'Word Application +Scope'". The file/format/install are not implicated. + +This maps to: +`references/activity-packages/word-activities/playbooks/read-text-missing-container.md` + +The correct agent behavior is to tie the fault to the missing container / +wrong Read Text surface and recommend either nesting it in a `Use Word File` +or switching to the standalone `System > File > Word Document` `Read Text` +(which takes a file path) — not blaming the file path, format, or Word +install. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a Word-pack `Read Text` loose in the sequence (no container, no file path) | + +> **Note on fixtures.** The faulted job's error names the `Read Text` +> activity and the missing `Use Word File` / `Word Application Scope` +> container; the `.xaml` shows the activity loose in the `Main Sequence`. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `read-text-missing-container.md` +- Agent identified the Word-pack Read Text being outside a `Use Word File` / + `Word Application Scope` as the cause and recommended nesting it in a + container OR switching to the standalone `System > File > Word Document` + `Read Text` — without blaming the file/format/install or fabricating + actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/28bf6cb314.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/28bf6cb314.json deleted file mode 100644 index 98d92180f5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/28bf6cb314.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f7a8b9c0-d1e2-4204-9c2d-81920314a5b6", - "State": "Faulted", - "ProcessName": "ContractTextExtract", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-10", - "MachineKey": "", - "StartTime": "2026-06-16T08:05:01.420Z", - "EndTime": "2026-06-16T08:05:02.380Z", - "CreationTime": "2026-06-16T08:05:01.210Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/36a7c93e47.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/36a7c93e47.json deleted file mode 100644 index 8cfc4eb06c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/36a7c93e47.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e6f7a8b9-c0d1-4193-8b1c-7081920314a5", - "Name": "Doc Extract", - "Path": "Doc Extract", - "Description": "Production folder for document text-extraction processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/README.md deleted file mode 100644 index 5dfabf0e8c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Read Text Failure - Activity Outside Its Container - -This scenario reproduces a `Read Text` failure where the modern Word-pack -`Read Text` activity is dropped **outside** a `Use Word File` / -`Word Application Scope` and faults as invalid — it has no file input of its -own and reads the container's open document. - -## What this scenario uncovers - -**Root Cause:** The Word-pack `Read Text` sits loose in the sequence with no -container. It reads the document held open by a surrounding scope; outside -one it is invalid, so the job faults immediately with "The 'Read Text' -activity must be placed inside a 'Use Word File' or 'Word Application -Scope'". The file/format/install are not implicated. - -This maps to: -`references/activity-packages/word-activities/playbooks/read-text-missing-container.md` - -The correct agent behavior is to tie the fault to the missing container / -wrong Read Text surface and recommend either nesting it in a `Use Word File` -or switching to the standalone `System > File > Word Document` `Read Text` -(which takes a file path) — not blaming the file path, format, or Word -install. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a Word-pack `Read Text` loose in the sequence (no container, no file path) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's error names the `Read Text` -> activity and the missing `Use Word File` / `Word Application Scope` -> container; the `.xaml` shows the activity loose in the `Main Sequence`. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `read-text-missing-container.md` -- Agent identified the Word-pack Read Text being outside a `Use Word File` / - `Word Application Scope` as the cause and recommended nesting it in a - container OR switching to the standalone `System > File > Word Document` - `Read Text` — without blaming the file/format/install or fabricating - actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/bda652460b.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/bda652460b.json deleted file mode 100644 index 695514eb5a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/bda652460b.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T08:05:02.310Z", - "Message": "Validation Error: The 'Read Text' activity must be placed inside a 'Use Word File' or 'Word Application Scope' container.\r\nin Main.xaml\r\n at ReadTextInDocument \"Read Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/c1ee1a29c7.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/c1ee1a29c7.json deleted file mode 100644 index 770467365e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/c1ee1a29c7.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f7a8b9c0-d1e2-4204-9c2d-81920314a5b6", - "StartTime": "2026-06-16T08:05:01.420Z", - "EndTime": "2026-06-16T08:05:02.380Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "a8b9c0d1-e2f3-4315-ad3e-920314a5b6c7", - "Info": "Validation Error: Activity 'Read Text' is not valid. The 'Read Text' activity must be placed inside a 'Use Word File' or 'Word Application Scope' container.\r\n\r\nin Main.xaml\r\n at ReadTextInDocument \"Read Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Word.Activities.WordException: The 'Read Text' activity must be used within a 'Use Word File' or 'Word Application Scope'.", - "CreationTime": "2026-06-16T08:05:01.210Z", - "ReleaseName": "ContractTextExtract", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-10", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021950, - "OrganizationUnitId": 7791618, - "FolderKey": "e6f7a8b9-c0d1-4193-8b1c-7081920314a5", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T08:05:02.380Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f7a8b9c0-d1e2-4204-9c2d-81920314a5b6", - "Id": 682455443 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/c97ff73448.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/c97ff73448.json deleted file mode 100644 index 0b16beb807..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/c97ff73448.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T08:05:01.470Z", - "Message": "Execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T08:05:02.310Z", - "Message": "Validation Error: The 'Read Text' activity must be placed inside a 'Use Word File' or 'Word Application Scope' container.\r\nin Main.xaml\r\n at ReadTextInDocument \"Read Text\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T08:05:02.360Z", - "Message": "Execution ended in: 00:00:00" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/manifest.json deleted file mode 100644 index 29b6d68930..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Read Text scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "36a7c93e47.json" - }, - { - "match": "or jobs list --folder-key \"e6f7a8b9-c0d1-4193-8b1c-7081920314a5\" --state Faulted --output json", - "file": "28bf6cb314.json" - }, - { - "match": "or jobs list --folder-key e6f7a8b9-c0d1-4193-8b1c-7081920314a5 --state Faulted --output json", - "file": "28bf6cb314.json" - }, - { - "match": "or jobs get \"f7a8b9c0-d1e2-4204-9c2d-81920314a5b6\" --output json", - "file": "c1ee1a29c7.json" - }, - { - "match": "or jobs get f7a8b9c0-d1e2-4204-9c2d-81920314a5b6 --output json", - "file": "c1ee1a29c7.json" - }, - { - "match": "or jobs logs \"f7a8b9c0-d1e2-4204-9c2d-81920314a5b6\" --level Error --output json", - "file": "bda652460b.json" - }, - { - "match": "or jobs logs f7a8b9c0-d1e2-4204-9c2d-81920314a5b6 --level Error --output json", - "file": "bda652460b.json" - }, - { - "match": "or jobs logs \"f7a8b9c0-d1e2-4204-9c2d-81920314a5b6\" --output json", - "file": "c97ff73448.json" - }, - { - "match": "or jobs logs f7a8b9c0-d1e2-4204-9c2d-81920314a5b6 --output json", - "file": "c97ff73448.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/uip-fixture.json new file mode 100644 index 0000000000..0f2e6f0fe4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e6f7a8b9-c0d1-4193-8b1c-7081920314a5", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7a8b9c0-d1e2-4204-9c2d-81920314a5b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ContractTextExtract\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-10\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T08:05:01.420Z\",\r\n \"EndTime\": \"2026-06-16T08:05:02.380Z\",\r\n \"CreationTime\": \"2026-06-16T08:05:01.210Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7a8b9c0-d1e2-4204-9c2d-81920314a5b6", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T08:05:02.310Z\",\r\n \"Message\": \"Validation Error: The 'Read Text' activity must be placed inside a 'Use Word File' or 'Word Application Scope' container.\\r\\nin Main.xaml\\r\\n at ReadTextInDocument \\\"Read Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f7a8b9c0-d1e2-4204-9c2d-81920314a5b6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f7a8b9c0-d1e2-4204-9c2d-81920314a5b6\",\r\n \"StartTime\": \"2026-06-16T08:05:01.420Z\",\r\n \"EndTime\": \"2026-06-16T08:05:02.380Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"a8b9c0d1-e2f3-4315-ad3e-920314a5b6c7\",\r\n \"Info\": \"Validation Error: Activity 'Read Text' is not valid. The 'Read Text' activity must be placed inside a 'Use Word File' or 'Word Application Scope' container.\\r\\n\\r\\nin Main.xaml\\r\\n at ReadTextInDocument \\\"Read Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Word.Activities.WordException: The 'Read Text' activity must be used within a 'Use Word File' or 'Word Application Scope'.\",\r\n \"CreationTime\": \"2026-06-16T08:05:01.210Z\",\r\n \"ReleaseName\": \"ContractTextExtract\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-10\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021950,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"e6f7a8b9-c0d1-4193-8b1c-7081920314a5\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T08:05:02.380Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f7a8b9c0-d1e2-4204-9c2d-81920314a5b6\",\r\n \"Id\": 682455443\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7a8b9c0-d1e2-4204-9c2d-81920314a5b6", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T08:05:01.470Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T08:05:02.310Z\",\r\n \"Message\": \"Validation Error: The 'Read Text' activity must be placed inside a 'Use Word File' or 'Word Application Scope' container.\\r\\nin Main.xaml\\r\\n at ReadTextInDocument \\\"Read Text\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T08:05:02.360Z\",\r\n \"Message\": \"Execution ended in: 00:00:00\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e6f7a8b9-c0d1-4193-8b1c-7081920314a5\",\r\n \"Name\": \"Doc Extract\",\r\n \"Path\": \"Doc Extract\",\r\n \"Description\": \"Production folder for document text-extraction processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/task.yaml index c877abc0e7..1a1f7db403 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-missing-container/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/README.md new file mode 100644 index 0000000000..71cb9110d2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/README.md @@ -0,0 +1,46 @@ +# Read Text Failure - Protected View Blocks an Externally-Sourced File + +This scenario reproduces a `Read Text` failure where Word opens an +email-sourced document (Mark-of-the-Web) in **Protected View**, blocking the +Interop read with a `COMException`. Internally-created documents read fine. + +## What this scenario uncovers + +**Root Cause:** The document arrived from email/internet, so it carries a +Mark-of-the-Web. Word opens it in Protected View (read-only sandbox) and the +Interop read is blocked. Locally-authored files have no Mark-of-the-Web and +read fine. This is a host file-trust / security configuration issue, not a +format, path, or install problem. + +This maps to: +`references/activity-packages/word-activities/playbooks/read-text-protected-view.md` + +The correct agent behavior is to tie the fault to Protected View / +Mark-of-the-Web on externally-sourced files and recommend unblocking the +file / Trusted Locations / disabling Protected View — framed as host steps +for the off-host user (not fabricating host actions, not blaming the format +or path). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Word Application Scope` + `Read Text` on an inbound email-attachment `.docx` | + +> **Note on fixtures.** The faulted job's `COMException` explicitly names +> Protected View and the "Internet/email location" origin; the user frames +> the failure as specific to externally-arriving files (internal docs read +> fine). The off-host framing means the agent must hand over host steps. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `read-text-protected-view.md` +- Agent identified Protected View / Mark-of-the-Web on the externally-sourced + file as the cause and recommended unblocking the file / Trusted Locations / + disabling Protected View as host steps — without fabricating host actions + or blaming the file format/path + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/476593df99.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/476593df99.json deleted file mode 100644 index d732184f61..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/476593df99.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-16T11:02:06.900Z", - "Message": "Word cannot open the document 'C:\\UiPath\\Inbound Docs\\data\\inbound\\Invoice_From_Email.docx' for reading because it is in Protected View (the file originated from an Internet/email location).\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/README.md deleted file mode 100644 index 28e28b5b98..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Read Text Failure - Protected View Blocks an Externally-Sourced File - -This scenario reproduces a `Read Text` failure where Word opens an -email-sourced document (Mark-of-the-Web) in **Protected View**, blocking the -Interop read with a `COMException`. Internally-created documents read fine. - -## What this scenario uncovers - -**Root Cause:** The document arrived from email/internet, so it carries a -Mark-of-the-Web. Word opens it in Protected View (read-only sandbox) and the -Interop read is blocked. Locally-authored files have no Mark-of-the-Web and -read fine. This is a host file-trust / security configuration issue, not a -format, path, or install problem. - -This maps to: -`references/activity-packages/word-activities/playbooks/read-text-protected-view.md` - -The correct agent behavior is to tie the fault to Protected View / -Mark-of-the-Web on externally-sourced files and recommend unblocking the -file / Trusted Locations / disabling Protected View — framed as host steps -for the off-host user (not fabricating host actions, not blaming the format -or path). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Word Application Scope` + `Read Text` on an inbound email-attachment `.docx` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's `COMException` explicitly names -> Protected View and the "Internet/email location" origin; the user frames -> the failure as specific to externally-arriving files (internal docs read -> fine). The off-host framing means the agent must hand over host steps. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `read-text-protected-view.md` -- Agent identified Protected View / Mark-of-the-Web on the externally-sourced - file as the cause and recommended unblocking the file / Trusted Locations / - disabling Protected View as host steps — without fabricating host actions - or blaming the file format/path diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/a1533bde0b.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/a1533bde0b.json deleted file mode 100644 index 4457aa1ded..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/a1533bde0b.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9", - "Name": "Inbound Docs", - "Path": "Inbound Docs", - "Description": "Production folder for inbound email-attachment processing.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/b7284bf8ec.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/b7284bf8ec.json deleted file mode 100644 index 85bdbd7196..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/b7284bf8ec.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0", - "StartTime": "2026-06-16T11:02:02.540Z", - "EndTime": "2026-06-16T11:02:06.980Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "e1f2a3b4-c5d6-4315-c172-36c7d8e9f001", - "Info": "Word Application Scope: Word cannot open the document 'C:\\UiPath\\Inbound Docs\\data\\inbound\\Invoice_From_Email.docx' for reading because it is in Protected View (the file originated from an Internet/email location).\r\n\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The command cannot be performed because a document is open in Protected View.", - "CreationTime": "2026-06-16T11:02:02.320Z", - "ReleaseName": "InboundDocReader", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6022170, - "OrganizationUnitId": 7791618, - "FolderKey": "c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-16T11:02:06.980Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0", - "Id": 682457665 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/e0ebddf14d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/e0ebddf14d.json deleted file mode 100644 index 53cc01b40c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/e0ebddf14d.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-16T11:02:02.590Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T11:02:04.310Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-16T11:02:05.520Z", - "Message": "[Word Application Scope] Opening 'data\\inbound\\Invoice_From_Email.docx'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-16T11:02:06.900Z", - "Message": "Word cannot open the document 'C:\\UiPath\\Inbound Docs\\data\\inbound\\Invoice_From_Email.docx' for reading because it is in Protected View (the file originated from an Internet/email location).\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-16T11:02:06.950Z", - "Message": "Execution ended in: 00:00:04" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/fa6a535125.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/fa6a535125.json deleted file mode 100644 index 0d803be129..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/fa6a535125.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0", - "State": "Faulted", - "ProcessName": "InboundDocReader", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-16T11:02:02.540Z", - "EndTime": "2026-06-16T11:02:06.980Z", - "CreationTime": "2026-06-16T11:02:02.320Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/manifest.json deleted file mode 100644 index 6e97bf25bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Read Text scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "a1533bde0b.json" - }, - { - "match": "or jobs list --folder-key \"c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9\" --state Faulted --output json", - "file": "fa6a535125.json" - }, - { - "match": "or jobs list --folder-key c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9 --state Faulted --output json", - "file": "fa6a535125.json" - }, - { - "match": "or jobs get \"d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0\" --output json", - "file": "b7284bf8ec.json" - }, - { - "match": "or jobs get d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0 --output json", - "file": "b7284bf8ec.json" - }, - { - "match": "or jobs logs \"d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0\" --level Error --output json", - "file": "476593df99.json" - }, - { - "match": "or jobs logs d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0 --level Error --output json", - "file": "476593df99.json" - }, - { - "match": "or jobs logs \"d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0\" --output json", - "file": "e0ebddf14d.json" - }, - { - "match": "or jobs logs d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0 --output json", - "file": "e0ebddf14d.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/uip-fixture.json new file mode 100644 index 0000000000..12d4b59df3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InboundDocReader\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-16T11:02:02.540Z\",\r\n \"EndTime\": \"2026-06-16T11:02:06.980Z\",\r\n \"CreationTime\": \"2026-06-16T11:02:02.320Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T11:02:06.900Z\",\r\n \"Message\": \"Word cannot open the document 'C:\\\\UiPath\\\\Inbound Docs\\\\data\\\\inbound\\\\Invoice_From_Email.docx' for reading because it is in Protected View (the file originated from an Internet/email location).\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0\",\r\n \"StartTime\": \"2026-06-16T11:02:02.540Z\",\r\n \"EndTime\": \"2026-06-16T11:02:06.980Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"e1f2a3b4-c5d6-4315-c172-36c7d8e9f001\",\r\n \"Info\": \"Word Application Scope: Word cannot open the document 'C:\\\\UiPath\\\\Inbound Docs\\\\data\\\\inbound\\\\Invoice_From_Email.docx' for reading because it is in Protected View (the file originated from an Internet/email location).\\r\\n\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The command cannot be performed because a document is open in Protected View.\",\r\n \"CreationTime\": \"2026-06-16T11:02:02.320Z\",\r\n \"ReleaseName\": \"InboundDocReader\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6022170,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-16T11:02:06.980Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0\",\r\n \"Id\": 682457665\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d0e1f2a3-b4c5-4204-c061-25b6c7d8e9f0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T11:02:02.590Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T11:02:04.310Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-16T11:02:05.520Z\",\r\n \"Message\": \"[Word Application Scope] Opening 'data\\\\inbound\\\\Invoice_From_Email.docx'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-16T11:02:06.900Z\",\r\n \"Message\": \"Word cannot open the document 'C:\\\\UiPath\\\\Inbound Docs\\\\data\\\\inbound\\\\Invoice_From_Email.docx' for reading because it is in Protected View (the file originated from an Internet/email location).\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-16T11:02:06.950Z\",\r\n \"Message\": \"Execution ended in: 00:00:04\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c9d0e1f2-a3b4-4193-bf50-14a5b6c7d8e9\",\r\n \"Name\": \"Inbound Docs\",\r\n \"Path\": \"Inbound Docs\",\r\n \"Description\": \"Production folder for inbound email-attachment processing.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/task.yaml index 868ea7e7cd..8c803bb33d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-read-text-protected-view/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/README.md new file mode 100644 index 0000000000..f0a8c64095 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/README.md @@ -0,0 +1,47 @@ +# Replace Text Failure - "Application is busy" / COM Interop Retry + +This scenario reproduces an intermittent `Replace Text` COM failure where +WINWORD.EXE is busy (a Word instance was already running on the host) and +the activity's COM call is rejected with `RPC_E_SERVERCALL_RETRYLATER` +(0x8001010A). + +## What this scenario uncovers + +**Root Cause:** Word was busy when the activity issued its COM call — the +job log shows a Word instance already running on the host (orphaned or +concurrent WINWORD.EXE). COM rejects the call; the intermittency +("re-running sometimes works") is the tell of a transient busy state, not a +code defect. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-com-busy.md` + +The correct agent behavior is to tie the `0x8001010A` HRESULT to a +busy/locked Word session and recommend clearing it (Kill Process WINWORD +before the scope, ensure not open elsewhere) and/or a Retry Scope — not to +attempt host commands itself (the user is off-host), and not to confuse it +with the scope-startup "Word not installed" COM error. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Replace Text` inside a `Word Application Scope` | + +> **Note on fixtures.** The full job log includes a "Word already running on +> the host" warning immediately before the `RPC_E_SERVERCALL_RETRYLATER` +> error — the signal that distinguishes a busy/locked instance from the +> scope-startup `REGDB_E_CLASSNOTREG` "Word not installed" fault. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-com-busy.md` +- Agent identified a busy/locked Word session (orphaned or concurrent + WINWORD.EXE) as the cause and recommended clearing it (Kill Process + WINWORD before the scope, not-open-elsewhere) and/or a Retry Scope — + without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/17a2e63318.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/17a2e63318.json deleted file mode 100644 index c581502075..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/17a2e63318.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162", - "StartTime": "2026-06-13T15:12:03.410Z", - "EndTime": "2026-06-13T15:12:07.260Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c4d5e6f7-a8b9-4315-0c1d-2e3f40516273", - "Info": "Replace Text: The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\r\n\r\nin Main.xaml\r\n at WordReplaceText \"Replace Text\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))", - "CreationTime": "2026-06-13T15:12:03.180Z", - "ReleaseName": "BatchLetterFill", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021188, - "OrganizationUnitId": 7791618, - "FolderKey": "a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-13T15:12:07.260Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162", - "Id": 682443319 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/1c46b4cc10.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/1c46b4cc10.json deleted file mode 100644 index ab77b1d701..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/1c46b4cc10.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162", - "State": "Faulted", - "ProcessName": "BatchLetterFill", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-13T15:12:03.410Z", - "EndTime": "2026-06-13T15:12:07.260Z", - "CreationTime": "2026-06-13T15:12:03.180Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/4dcb8ac841.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/4dcb8ac841.json deleted file mode 100644 index 545ae0a373..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/4dcb8ac841.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-13T15:12:03.460Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T15:12:05.020Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-13T15:12:06.300Z", - "Message": "[Word Application Scope] A Microsoft Word instance was already running on the host when the scope started." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T15:12:07.190Z", - "Message": "The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\r\nin Main.xaml\r\n at WordReplaceText \"Replace Text\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T15:12:07.250Z", - "Message": "Execution ended in: 00:00:03" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/6ce601f459.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/6ce601f459.json deleted file mode 100644 index e2196a42ae..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/6ce601f459.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051", - "Name": "Doc Batch", - "Path": "Doc Batch", - "Description": "Production folder for batch document generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/70b8bca898.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/70b8bca898.json deleted file mode 100644 index b94630fd5f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/70b8bca898.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-13T15:12:07.190Z", - "Message": "The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\r\nin Main.xaml\r\n at WordReplaceText \"Replace Text\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/README.md deleted file mode 100644 index 73ef6e09c1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Replace Text Failure - "Application is busy" / COM Interop Retry - -This scenario reproduces an intermittent `Replace Text` COM failure where -WINWORD.EXE is busy (a Word instance was already running on the host) and -the activity's COM call is rejected with `RPC_E_SERVERCALL_RETRYLATER` -(0x8001010A). - -## What this scenario uncovers - -**Root Cause:** Word was busy when the activity issued its COM call — the -job log shows a Word instance already running on the host (orphaned or -concurrent WINWORD.EXE). COM rejects the call; the intermittency -("re-running sometimes works") is the tell of a transient busy state, not a -code defect. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-com-busy.md` - -The correct agent behavior is to tie the `0x8001010A` HRESULT to a -busy/locked Word session and recommend clearing it (Kill Process WINWORD -before the scope, ensure not open elsewhere) and/or a Retry Scope — not to -attempt host commands itself (the user is off-host), and not to confuse it -with the scope-startup "Word not installed" COM error. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Replace Text` inside a `Word Application Scope` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The full job log includes a "Word already running on -> the host" warning immediately before the `RPC_E_SERVERCALL_RETRYLATER` -> error — the signal that distinguishes a busy/locked instance from the -> scope-startup `REGDB_E_CLASSNOTREG` "Word not installed" fault. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-com-busy.md` -- Agent identified a busy/locked Word session (orphaned or concurrent - WINWORD.EXE) as the cause and recommended clearing it (Kill Process - WINWORD before the scope, not-open-elsewhere) and/or a Retry Scope — - without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/manifest.json deleted file mode 100644 index d24be2c2ab..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Replace Text scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6ce601f459.json" - }, - { - "match": "or jobs list --folder-key \"a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051\" --state Faulted --output json", - "file": "1c46b4cc10.json" - }, - { - "match": "or jobs list --folder-key a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051 --state Faulted --output json", - "file": "1c46b4cc10.json" - }, - { - "match": "or jobs get \"b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162\" --output json", - "file": "17a2e63318.json" - }, - { - "match": "or jobs get b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162 --output json", - "file": "17a2e63318.json" - }, - { - "match": "or jobs logs \"b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162\" --level Error --output json", - "file": "70b8bca898.json" - }, - { - "match": "or jobs logs b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162 --level Error --output json", - "file": "70b8bca898.json" - }, - { - "match": "or jobs logs \"b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162\" --output json", - "file": "4dcb8ac841.json" - }, - { - "match": "or jobs logs b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162 --output json", - "file": "4dcb8ac841.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/uip-fixture.json new file mode 100644 index 0000000000..e0bbf743e4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BatchLetterFill\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-13T15:12:03.410Z\",\r\n \"EndTime\": \"2026-06-13T15:12:07.260Z\",\r\n \"CreationTime\": \"2026-06-13T15:12:03.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T15:12:07.190Z\",\r\n \"Message\": \"The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\\r\\nin Main.xaml\\r\\n at WordReplaceText \\\"Replace Text\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162\",\r\n \"StartTime\": \"2026-06-13T15:12:03.410Z\",\r\n \"EndTime\": \"2026-06-13T15:12:07.260Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c4d5e6f7-a8b9-4315-0c1d-2e3f40516273\",\r\n \"Info\": \"Replace Text: The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\\r\\n\\r\\nin Main.xaml\\r\\n at WordReplaceText \\\"Replace Text\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\",\r\n \"CreationTime\": \"2026-06-13T15:12:03.180Z\",\r\n \"ReleaseName\": \"BatchLetterFill\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021188,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-13T15:12:07.260Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162\",\r\n \"Id\": 682443319\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b3c4d5e6-f7a8-4204-9b0c-1d2e3f405162", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T15:12:03.460Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T15:12:05.020Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-13T15:12:06.300Z\",\r\n \"Message\": \"[Word Application Scope] A Microsoft Word instance was already running on the host when the scope started.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T15:12:07.190Z\",\r\n \"Message\": \"The message filter indicated that the application is busy. (Exception from HRESULT: 0x8001010A (RPC_E_SERVERCALL_RETRYLATER))\\r\\nin Main.xaml\\r\\n at WordReplaceText \\\"Replace Text\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T15:12:07.250Z\",\r\n \"Message\": \"Execution ended in: 00:00:03\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a2b3c4d5-e6f7-4193-8a9b-0c1d2e3f4051\",\r\n \"Name\": \"Doc Batch\",\r\n \"Path\": \"Doc Batch\",\r\n \"Description\": \"Production folder for batch document generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/task.yaml index 132182f437..e15cdb238d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-com-busy/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/README.md new file mode 100644 index 0000000000..0f0b8d7b09 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/README.md @@ -0,0 +1,49 @@ +# Replace Text Failure - File Lock / "Being Used By Another Process" + +This scenario reproduces a `Replace Text` file-lock failure: a `Use Word +File` scope with **Auto Save** enabled edits the same shared file on every +iteration of a `For Each` loop, so a save races the next iteration's access +and faults with `System.IO.IOException: ... because it is being used by +another process`. + +## What this scenario uncovers + +**Root Cause:** `Use Word File` with `AutoSave="True"` targets one shared +file (`MasterContract.docx`) inside a loop. Auto Save persists the document +while the next iteration is already opening it, so the save races another +access and locks. The "first item works, then it dies" pattern plus the +per-iteration Auto-Save/re-open logs are the tell — not a corrupt file, a +missing install, or a path problem. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-file-locked.md` + +The correct agent behavior is to tie the `IOException` to the Auto-Save + +loop-on-a-shared-file contention and recommend unchecking Auto Save, writing +a distinct output path per iteration, or clearing a concurrent accessor — +not blaming corruption, Word install, or the path. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Use Word File` (`AutoSave="True"`) + `Replace Text in Document` inside a `For Each` over one shared file | + +> **Note on fixtures.** The full job log shows per-iteration trace lines +> (`Iteration 1: Auto Save writing ...` -> `Iteration 2: opened ...` -> +> IOException), exposing the save-vs-access race on the single shared file. +> Distinct from `word-scope-file-corrupted` (corrupted-on-open from a +> half-written template). + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-file-locked.md` +- Agent identified the Auto-Save + loop-on-a-shared-file lock as the cause + and recommended unchecking Auto Save, writing a distinct path per + iteration, or clearing the concurrent accessor — without blaming + corruption / install / path or fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/2d5c928b85.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/2d5c928b85.json deleted file mode 100644 index 9b16044d1d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/2d5c928b85.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-14T13:30:02.750Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T13:30:04.260Z", - "Message": "[For Each recipient] Iteration 1: opened 'data\\MasterContract.docx'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T13:30:06.540Z", - "Message": "[For Each recipient] Iteration 1: Auto Save writing 'data\\MasterContract.docx'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T13:30:08.110Z", - "Message": "[For Each recipient] Iteration 2: opened 'data\\MasterContract.docx'" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-14T13:30:09.380Z", - "Message": "The process cannot access the file 'C:\\UiPath\\Contracts Batch\\data\\MasterContract.docx' because it is being used by another process.\r\nin Main.xaml\r\n at ReplaceTextInDocument \"Replace Text in Document\"\r\n at WordApplicationCard \"Use Word File\"\r\n at ForEach \"For Each recipient\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T13:30:09.440Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/5d1ecb5489.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/5d1ecb5489.json deleted file mode 100644 index e067d7608d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/5d1ecb5489.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f7a8b9c0-d1e2-4204-3f40-516273849506", - "StartTime": "2026-06-14T13:30:02.700Z", - "EndTime": "2026-06-14T13:30:09.450Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "a8b9c0d1-e2f3-4315-3f40-516273849517", - "Info": "Use Word File: The process cannot access the file 'C:\\UiPath\\Contracts Batch\\data\\MasterContract.docx' because it is being used by another process.\r\n\r\nin Main.xaml\r\n at ReplaceTextInDocument \"Replace Text in Document\"\r\n at WordApplicationCard \"Use Word File\"\r\n at ForEach \"For Each recipient\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.IOException: The process cannot access the file 'C:\\UiPath\\Contracts Batch\\data\\MasterContract.docx' because it is being used by another process.", - "CreationTime": "2026-06-14T13:30:02.460Z", - "ReleaseName": "ContractMerge", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-06", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021455, - "OrganizationUnitId": 7791618, - "FolderKey": "e6f7a8b9-c0d1-4193-2e3f-405162738495", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-14T13:30:09.450Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f7a8b9c0-d1e2-4204-3f40-516273849506", - "Id": 682451890 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/README.md deleted file mode 100644 index baaa8113b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Replace Text Failure - File Lock / "Being Used By Another Process" - -This scenario reproduces a `Replace Text` file-lock failure: a `Use Word -File` scope with **Auto Save** enabled edits the same shared file on every -iteration of a `For Each` loop, so a save races the next iteration's access -and faults with `System.IO.IOException: ... because it is being used by -another process`. - -## What this scenario uncovers - -**Root Cause:** `Use Word File` with `AutoSave="True"` targets one shared -file (`MasterContract.docx`) inside a loop. Auto Save persists the document -while the next iteration is already opening it, so the save races another -access and locks. The "first item works, then it dies" pattern plus the -per-iteration Auto-Save/re-open logs are the tell — not a corrupt file, a -missing install, or a path problem. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-file-locked.md` - -The correct agent behavior is to tie the `IOException` to the Auto-Save + -loop-on-a-shared-file contention and recommend unchecking Auto Save, writing -a distinct output path per iteration, or clearing a concurrent accessor — -not blaming corruption, Word install, or the path. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Use Word File` (`AutoSave="True"`) + `Replace Text in Document` inside a `For Each` over one shared file | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The full job log shows per-iteration trace lines -> (`Iteration 1: Auto Save writing ...` -> `Iteration 2: opened ...` -> -> IOException), exposing the save-vs-access race on the single shared file. -> Distinct from `word-scope-file-corrupted` (corrupted-on-open from a -> half-written template). - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-file-locked.md` -- Agent identified the Auto-Save + loop-on-a-shared-file lock as the cause - and recommended unchecking Auto Save, writing a distinct path per - iteration, or clearing the concurrent accessor — without blaming - corruption / install / path or fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/abb6d91052.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/abb6d91052.json deleted file mode 100644 index b324a80d28..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/abb6d91052.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "e6f7a8b9-c0d1-4193-2e3f-405162738495", - "Name": "Contracts Batch", - "Path": "Contracts Batch", - "Description": "Production folder for batch contract merge processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/b75f5731f6.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/b75f5731f6.json deleted file mode 100644 index a1abbc6371..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/b75f5731f6.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "f7a8b9c0-d1e2-4204-3f40-516273849506", - "State": "Faulted", - "ProcessName": "ContractMerge", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-06", - "MachineKey": "", - "StartTime": "2026-06-14T13:30:02.700Z", - "EndTime": "2026-06-14T13:30:09.450Z", - "CreationTime": "2026-06-14T13:30:02.460Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/da974ec4a8.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/da974ec4a8.json deleted file mode 100644 index a636faa962..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/da974ec4a8.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-14T13:30:09.380Z", - "Message": "The process cannot access the file 'C:\\UiPath\\Contracts Batch\\data\\MasterContract.docx' because it is being used by another process.\r\nin Main.xaml\r\n at ReplaceTextInDocument \"Replace Text in Document\"\r\n at WordApplicationCard \"Use Word File\"\r\n at ForEach \"For Each recipient\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/manifest.json deleted file mode 100644 index e29ad5e138..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Replace Text scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "abb6d91052.json" - }, - { - "match": "or jobs list --folder-key \"e6f7a8b9-c0d1-4193-2e3f-405162738495\" --state Faulted --output json", - "file": "b75f5731f6.json" - }, - { - "match": "or jobs list --folder-key e6f7a8b9-c0d1-4193-2e3f-405162738495 --state Faulted --output json", - "file": "b75f5731f6.json" - }, - { - "match": "or jobs get \"f7a8b9c0-d1e2-4204-3f40-516273849506\" --output json", - "file": "5d1ecb5489.json" - }, - { - "match": "or jobs get f7a8b9c0-d1e2-4204-3f40-516273849506 --output json", - "file": "5d1ecb5489.json" - }, - { - "match": "or jobs logs \"f7a8b9c0-d1e2-4204-3f40-516273849506\" --level Error --output json", - "file": "da974ec4a8.json" - }, - { - "match": "or jobs logs f7a8b9c0-d1e2-4204-3f40-516273849506 --level Error --output json", - "file": "da974ec4a8.json" - }, - { - "match": "or jobs logs \"f7a8b9c0-d1e2-4204-3f40-516273849506\" --output json", - "file": "2d5c928b85.json" - }, - { - "match": "or jobs logs f7a8b9c0-d1e2-4204-3f40-516273849506 --output json", - "file": "2d5c928b85.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/uip-fixture.json new file mode 100644 index 0000000000..761a573d9d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e6f7a8b9-c0d1-4193-2e3f-405162738495", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f7a8b9c0-d1e2-4204-3f40-516273849506\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ContractMerge\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-06\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-14T13:30:02.700Z\",\r\n \"EndTime\": \"2026-06-14T13:30:09.450Z\",\r\n \"CreationTime\": \"2026-06-14T13:30:02.460Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7a8b9c0-d1e2-4204-3f40-516273849506", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T13:30:09.380Z\",\r\n \"Message\": \"The process cannot access the file 'C:\\\\UiPath\\\\Contracts Batch\\\\data\\\\MasterContract.docx' because it is being used by another process.\\r\\nin Main.xaml\\r\\n at ReplaceTextInDocument \\\"Replace Text in Document\\\"\\r\\n at WordApplicationCard \\\"Use Word File\\\"\\r\\n at ForEach \\\"For Each recipient\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f7a8b9c0-d1e2-4204-3f40-516273849506", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f7a8b9c0-d1e2-4204-3f40-516273849506\",\r\n \"StartTime\": \"2026-06-14T13:30:02.700Z\",\r\n \"EndTime\": \"2026-06-14T13:30:09.450Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"a8b9c0d1-e2f3-4315-3f40-516273849517\",\r\n \"Info\": \"Use Word File: The process cannot access the file 'C:\\\\UiPath\\\\Contracts Batch\\\\data\\\\MasterContract.docx' because it is being used by another process.\\r\\n\\r\\nin Main.xaml\\r\\n at ReplaceTextInDocument \\\"Replace Text in Document\\\"\\r\\n at WordApplicationCard \\\"Use Word File\\\"\\r\\n at ForEach \\\"For Each recipient\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.IOException: The process cannot access the file 'C:\\\\UiPath\\\\Contracts Batch\\\\data\\\\MasterContract.docx' because it is being used by another process.\",\r\n \"CreationTime\": \"2026-06-14T13:30:02.460Z\",\r\n \"ReleaseName\": \"ContractMerge\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-06\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021455,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"e6f7a8b9-c0d1-4193-2e3f-405162738495\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-14T13:30:09.450Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f7a8b9c0-d1e2-4204-3f40-516273849506\",\r\n \"Id\": 682451890\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f7a8b9c0-d1e2-4204-3f40-516273849506", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T13:30:02.750Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T13:30:04.260Z\",\r\n \"Message\": \"[For Each recipient] Iteration 1: opened 'data\\\\MasterContract.docx'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T13:30:06.540Z\",\r\n \"Message\": \"[For Each recipient] Iteration 1: Auto Save writing 'data\\\\MasterContract.docx'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T13:30:08.110Z\",\r\n \"Message\": \"[For Each recipient] Iteration 2: opened 'data\\\\MasterContract.docx'\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-14T13:30:09.380Z\",\r\n \"Message\": \"The process cannot access the file 'C:\\\\UiPath\\\\Contracts Batch\\\\data\\\\MasterContract.docx' because it is being used by another process.\\r\\nin Main.xaml\\r\\n at ReplaceTextInDocument \\\"Replace Text in Document\\\"\\r\\n at WordApplicationCard \\\"Use Word File\\\"\\r\\n at ForEach \\\"For Each recipient\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T13:30:09.440Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e6f7a8b9-c0d1-4193-2e3f-405162738495\",\r\n \"Name\": \"Contracts Batch\",\r\n \"Path\": \"Contracts Batch\",\r\n \"Description\": \"Production folder for batch contract merge processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/task.yaml index 1e48d03c32..1fc40acbd2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-file-locked/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/README.md new file mode 100644 index 0000000000..025e8cef00 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/README.md @@ -0,0 +1,48 @@ +# Replace Text Failure - Headers / Footers / Text Boxes Skipped + +This scenario reproduces a `Replace Text` silent miss where the +`[CompanyName]` placeholder in the document **header/footer** is left +unchanged while the **body** copy is replaced, because the project pins an +older `UiPath.Word.Activities` (1.6.0) that scans only body text. The job +completes **Successfully**. + +## What this scenario uncovers + +**Root Cause:** Older `UiPath.Word.Activities` iterates only the primary +body story range; headers, footers, and floating text boxes are skipped. The +body `[CompanyName]` is replaced (log: "Replaced 1 occurrence in the document +body") while the header copy survives. The body match proves the Search +value and template are correct — the gap is the package version. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-headers-textboxes-skipped.md` + +The correct agent behavior is to recognize a **Successful job with the +header unreplaced** as a silent failure, tie it to the old package version / +non-body story range (via the "in the document body" log + `project.json` +pin), and recommend updating `UiPath.Word.Activities` — not blaming the +template/Search value or dismissing the green job. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project pinning `UiPath.Word.Activities [1.6.0]`; `Replace Text` on `[CompanyName]` | + +> **Note on fixtures.** No faulted job (State=Successful) and no error logs; +> the trace explicitly says the replacement happened "in the document body", +> and `project.json` pins the old `[1.6.0]` package. The agent diagnoses from +> the body-only log + the version pin + the user's header symptom. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-headers-textboxes-skipped.md` +- Agent identified the older `UiPath.Word.Activities` scanning only body text + as the cause and recommended updating the package (or addressing the header + range explicitly / in code) — without treating the green job as healthy or + blaming the template/Search value + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/28a795b76f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/28a795b76f.json deleted file mode 100644 index 0fbc229aad..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/28a795b76f.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c3d4e5f6-a7b8-4204-9900-4d5e6f708192", - "State": "Successful", - "ProcessName": "BrandedLetterFill", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-08", - "MachineKey": "", - "StartTime": "2026-06-15T09:22:01.330Z", - "EndTime": "2026-06-15T09:22:07.910Z", - "CreationTime": "2026-06-15T09:22:01.110Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/361f107741.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/361f107741.json deleted file mode 100644 index f6fb635e62..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/361f107741.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081", - "Name": "Branded Docs", - "Path": "Branded Docs", - "Description": "Production folder for branded document generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/4ec4622d5a.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/4ec4622d5a.json deleted file mode 100644 index 41b4e75538..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/4ec4622d5a.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T09:22:01.380Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T09:22:03.140Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T09:22:04.560Z", - "Message": "[Word Application Scope] Document 'data\\BrandedLetter.docx' opened" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T09:22:05.880Z", - "Message": "[Replace Text] Executed. Replaced 1 occurrence(s) of '[CompanyName]' in the document body." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T09:22:07.880Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/README.md deleted file mode 100644 index d9b25adc0d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Replace Text Failure - Headers / Footers / Text Boxes Skipped - -This scenario reproduces a `Replace Text` silent miss where the -`[CompanyName]` placeholder in the document **header/footer** is left -unchanged while the **body** copy is replaced, because the project pins an -older `UiPath.Word.Activities` (1.6.0) that scans only body text. The job -completes **Successfully**. - -## What this scenario uncovers - -**Root Cause:** Older `UiPath.Word.Activities` iterates only the primary -body story range; headers, footers, and floating text boxes are skipped. The -body `[CompanyName]` is replaced (log: "Replaced 1 occurrence in the document -body") while the header copy survives. The body match proves the Search -value and template are correct — the gap is the package version. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-headers-textboxes-skipped.md` - -The correct agent behavior is to recognize a **Successful job with the -header unreplaced** as a silent failure, tie it to the old package version / -non-body story range (via the "in the document body" log + `project.json` -pin), and recommend updating `UiPath.Word.Activities` — not blaming the -template/Search value or dismissing the green job. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project pinning `UiPath.Word.Activities [1.6.0]`; `Replace Text` on `[CompanyName]` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** No faulted job (State=Successful) and no error logs; -> the trace explicitly says the replacement happened "in the document body", -> and `project.json` pins the old `[1.6.0]` package. The agent diagnoses from -> the body-only log + the version pin + the user's header symptom. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-headers-textboxes-skipped.md` -- Agent identified the older `UiPath.Word.Activities` scanning only body text - as the cause and recommended updating the package (or addressing the header - range explicitly / in code) — without treating the green job as healthy or - blaming the template/Search value diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/d9c28e43d0.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/d9c28e43d0.json deleted file mode 100644 index b7afe4bb08..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/d9c28e43d0.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3d4e5f6-a7b8-4204-9900-4d5e6f708192", - "StartTime": "2026-06-15T09:22:01.330Z", - "EndTime": "2026-06-15T09:22:07.910Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d4e5f6a7-b8c9-4315-aa11-5e6f70819203", - "Info": "The job completed successfully.", - "CreationTime": "2026-06-15T09:22:01.110Z", - "ReleaseName": "BrandedLetterFill", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-08", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021730, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T09:22:07.910Z", - "JobError": null, - "ErrorCode": "", - "TraceId": "c3d4e5f6-a7b8-4204-9900-4d5e6f708192", - "Id": 682453221 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/manifest.json deleted file mode 100644 index 5953177e94..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Replace Text scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "361f107741.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081 --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081\" --state Successful --output json", - "file": "28a795b76f.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081 --state Successful --output json", - "file": "28a795b76f.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081\" --output json", - "file": "28a795b76f.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081 --output json", - "file": "28a795b76f.json" - }, - { - "match": "or jobs get \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\" --output json", - "file": "d9c28e43d0.json" - }, - { - "match": "or jobs get c3d4e5f6-a7b8-4204-9900-4d5e6f708192 --output json", - "file": "d9c28e43d0.json" - }, - { - "match": "or jobs logs \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs c3d4e5f6-a7b8-4204-9900-4d5e6f708192 --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\" --output json", - "file": "4ec4622d5a.json" - }, - { - "match": "or jobs logs c3d4e5f6-a7b8-4204-9900-4d5e6f708192 --output json", - "file": "4ec4622d5a.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/uip-fixture.json new file mode 100644 index 0000000000..674d81c573 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BrandedLetterFill\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-08\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T09:22:01.330Z\",\r\n \"EndTime\": \"2026-06-15T09:22:07.910Z\",\r\n \"CreationTime\": \"2026-06-15T09:22:01.110Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3d4e5f6-a7b8-4204-9900-4d5e6f708192", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BrandedLetterFill\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-08\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T09:22:01.330Z\",\r\n \"EndTime\": \"2026-06-15T09:22:07.910Z\",\r\n \"CreationTime\": \"2026-06-15T09:22:01.110Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3d4e5f6-a7b8-4204-9900-4d5e6f708192", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\",\r\n \"StartTime\": \"2026-06-15T09:22:01.330Z\",\r\n \"EndTime\": \"2026-06-15T09:22:07.910Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d4e5f6a7-b8c9-4315-aa11-5e6f70819203\",\r\n \"Info\": \"The job completed successfully.\",\r\n \"CreationTime\": \"2026-06-15T09:22:01.110Z\",\r\n \"ReleaseName\": \"BrandedLetterFill\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-08\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021730,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T09:22:07.910Z\",\r\n \"JobError\": null,\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"c3d4e5f6-a7b8-4204-9900-4d5e6f708192\",\r\n \"Id\": 682453221\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3d4e5f6-a7b8-4204-9900-4d5e6f708192", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T09:22:01.380Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T09:22:03.140Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T09:22:04.560Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\BrandedLetter.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T09:22:05.880Z\",\r\n \"Message\": \"[Replace Text] Executed. Replaced 1 occurrence(s) of '[CompanyName]' in the document body.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T09:22:07.880Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-4193-8809-3c4d5e6f7081\",\r\n \"Name\": \"Branded Docs\",\r\n \"Path\": \"Branded Docs\",\r\n \"Description\": \"Production folder for branded document generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/task.yaml index dc58144024..9af5f303aa 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-headers-skipped/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/README.md new file mode 100644 index 0000000000..d89085cc9f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/README.md @@ -0,0 +1,44 @@ +# Replace Text Failure - 256-Character Input Limit + +This scenario reproduces a classic `Replace Text` (`WordReplaceText`) +failure where the `Replace` value exceeds the classic activity's hard +256-character limit, faulting with `System.ArgumentException: Value cannot +be longer than 256 characters`. + +## What this scenario uncovers + +**Root Cause:** Classic `Replace Text` caps `Search`/`Replace` inputs at +256 characters. The replacement clause is longer, so the activity throws +`ArgumentException` on the `Replace` parameter. Shorter clauses pass; the +project pins an older `UiPath.Word.Activities` version where the cap +applies. The document opens fine — the fault is purely the input length. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-length-limit.md` + +The correct agent behavior is to tie the `ArgumentException` to the 256-char +classic cap and recommend either upgrading `UiPath.Word.Activities` or doing +the substitution via `String.Replace` / regex in code — not editing the +template or blaming Word. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; classic `Replace Text` with a long `Replace` value, pinning `UiPath.Word.Activities [1.7.0]` | + +> **Note on fixtures.** The logs show the document opening successfully +> (`Document ... opened`) before the `ArgumentException`, locating the fault +> at the substitution input rather than at file access or COM. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-length-limit.md` +- Agent identified the 256-character classic input limit as the cause and + recommended upgrading the package or substituting via string/regex in code + — without editing the template or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/42779b0bd2.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/42779b0bd2.json deleted file mode 100644 index e42b47eef2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/42779b0bd2.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e1f2a3b4-c5d6-4204-e8f9-012435465718", - "StartTime": "2026-06-13T11:40:02.330Z", - "EndTime": "2026-06-13T11:40:05.910Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "f2a3b4c5-d6e7-4315-f901-243546571829", - "Info": "Replace Text: Value cannot be longer than 256 characters. (Parameter 'Replace')\r\n\r\nin Main.xaml\r\n at WordReplaceText \"Replace Text\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: Value cannot be longer than 256 characters. (Parameter 'Replace')", - "CreationTime": "2026-06-13T11:40:02.080Z", - "ReleaseName": "TermsLetterGen", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-04", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021033, - "OrganizationUnitId": 7791618, - "FolderKey": "d0e1f2a3-b4c5-4193-d7e8-f90124354607", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-13T11:40:05.910Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e1f2a3b4-c5d6-4204-e8f9-012435465718", - "Id": 682438876 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/8ccd87c0a2.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/8ccd87c0a2.json deleted file mode 100644 index c50330246f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/8ccd87c0a2.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-13T11:40:02.380Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T11:40:04.110Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-13T11:40:05.420Z", - "Message": "[Word Application Scope] Document 'data\\TermsTemplate.docx' opened" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-13T11:40:05.840Z", - "Message": "Value cannot be longer than 256 characters. (Parameter 'Replace')\r\nin Main.xaml\r\n at WordReplaceText \"Replace Text\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-13T11:40:05.900Z", - "Message": "Execution ended in: 00:00:03" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/94c3266c64.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/94c3266c64.json deleted file mode 100644 index 5d671bc092..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/94c3266c64.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d0e1f2a3-b4c5-4193-d7e8-f90124354607", - "Name": "Mail Merge", - "Path": "Mail Merge", - "Description": "Production folder for document mail-merge processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/README.md deleted file mode 100644 index a5ae086f21..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/README.md +++ /dev/null @@ -1,43 +0,0 @@ -# Replace Text Failure - 256-Character Input Limit - -This scenario reproduces a classic `Replace Text` (`WordReplaceText`) -failure where the `Replace` value exceeds the classic activity's hard -256-character limit, faulting with `System.ArgumentException: Value cannot -be longer than 256 characters`. - -## What this scenario uncovers - -**Root Cause:** Classic `Replace Text` caps `Search`/`Replace` inputs at -256 characters. The replacement clause is longer, so the activity throws -`ArgumentException` on the `Replace` parameter. Shorter clauses pass; the -project pins an older `UiPath.Word.Activities` version where the cap -applies. The document opens fine — the fault is purely the input length. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-length-limit.md` - -The correct agent behavior is to tie the `ArgumentException` to the 256-char -classic cap and recommend either upgrading `UiPath.Word.Activities` or doing -the substitution via `String.Replace` / regex in code — not editing the -template or blaming Word. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; classic `Replace Text` with a long `Replace` value, pinning `UiPath.Word.Activities [1.7.0]` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The logs show the document opening successfully -> (`Document ... opened`) before the `ArgumentException`, locating the fault -> at the substitution input rather than at file access or COM. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-length-limit.md` -- Agent identified the 256-character classic input limit as the cause and - recommended upgrading the package or substituting via string/regex in code - — without editing the template or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/a69bde7e56.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/a69bde7e56.json deleted file mode 100644 index 5f54dc3ba5..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/a69bde7e56.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-13T11:40:05.840Z", - "Message": "Value cannot be longer than 256 characters. (Parameter 'Replace')\r\nin Main.xaml\r\n at WordReplaceText \"Replace Text\"\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/d391bfcb29.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/d391bfcb29.json deleted file mode 100644 index 37f73524ce..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/d391bfcb29.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e1f2a3b4-c5d6-4204-e8f9-012435465718", - "State": "Faulted", - "ProcessName": "TermsLetterGen", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-04", - "MachineKey": "", - "StartTime": "2026-06-13T11:40:02.330Z", - "EndTime": "2026-06-13T11:40:05.910Z", - "CreationTime": "2026-06-13T11:40:02.080Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/manifest.json deleted file mode 100644 index 8702521c67..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Replace Text length-limit scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "94c3266c64.json" - }, - { - "match": "or jobs list --folder-key \"d0e1f2a3-b4c5-4193-d7e8-f90124354607\" --state Faulted --output json", - "file": "d391bfcb29.json" - }, - { - "match": "or jobs list --folder-key d0e1f2a3-b4c5-4193-d7e8-f90124354607 --state Faulted --output json", - "file": "d391bfcb29.json" - }, - { - "match": "or jobs get \"e1f2a3b4-c5d6-4204-e8f9-012435465718\" --output json", - "file": "42779b0bd2.json" - }, - { - "match": "or jobs get e1f2a3b4-c5d6-4204-e8f9-012435465718 --output json", - "file": "42779b0bd2.json" - }, - { - "match": "or jobs logs \"e1f2a3b4-c5d6-4204-e8f9-012435465718\" --level Error --output json", - "file": "a69bde7e56.json" - }, - { - "match": "or jobs logs e1f2a3b4-c5d6-4204-e8f9-012435465718 --level Error --output json", - "file": "a69bde7e56.json" - }, - { - "match": "or jobs logs \"e1f2a3b4-c5d6-4204-e8f9-012435465718\" --output json", - "file": "8ccd87c0a2.json" - }, - { - "match": "or jobs logs e1f2a3b4-c5d6-4204-e8f9-012435465718 --output json", - "file": "8ccd87c0a2.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/uip-fixture.json new file mode 100644 index 0000000000..8f9644dcc2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d0e1f2a3-b4c5-4193-d7e8-f90124354607", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e1f2a3b4-c5d6-4204-e8f9-012435465718\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TermsLetterGen\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-04\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-13T11:40:02.330Z\",\r\n \"EndTime\": \"2026-06-13T11:40:05.910Z\",\r\n \"CreationTime\": \"2026-06-13T11:40:02.080Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e1f2a3b4-c5d6-4204-e8f9-012435465718", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T11:40:05.840Z\",\r\n \"Message\": \"Value cannot be longer than 256 characters. (Parameter 'Replace')\\r\\nin Main.xaml\\r\\n at WordReplaceText \\\"Replace Text\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e1f2a3b4-c5d6-4204-e8f9-012435465718", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e1f2a3b4-c5d6-4204-e8f9-012435465718\",\r\n \"StartTime\": \"2026-06-13T11:40:02.330Z\",\r\n \"EndTime\": \"2026-06-13T11:40:05.910Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"f2a3b4c5-d6e7-4315-f901-243546571829\",\r\n \"Info\": \"Replace Text: Value cannot be longer than 256 characters. (Parameter 'Replace')\\r\\n\\r\\nin Main.xaml\\r\\n at WordReplaceText \\\"Replace Text\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: Value cannot be longer than 256 characters. (Parameter 'Replace')\",\r\n \"CreationTime\": \"2026-06-13T11:40:02.080Z\",\r\n \"ReleaseName\": \"TermsLetterGen\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-04\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021033,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"d0e1f2a3-b4c5-4193-d7e8-f90124354607\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-13T11:40:05.910Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e1f2a3b4-c5d6-4204-e8f9-012435465718\",\r\n \"Id\": 682438876\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e1f2a3b4-c5d6-4204-e8f9-012435465718", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T11:40:02.380Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T11:40:04.110Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-13T11:40:05.420Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\TermsTemplate.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-13T11:40:05.840Z\",\r\n \"Message\": \"Value cannot be longer than 256 characters. (Parameter 'Replace')\\r\\nin Main.xaml\\r\\n at WordReplaceText \\\"Replace Text\\\"\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-13T11:40:05.900Z\",\r\n \"Message\": \"Execution ended in: 00:00:03\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d0e1f2a3-b4c5-4193-d7e8-f90124354607\",\r\n \"Name\": \"Mail Merge\",\r\n \"Path\": \"Mail Merge\",\r\n \"Description\": \"Production folder for document mail-merge processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/task.yaml index f662d4d263..8d5a56a8aa 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-length-limit/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/README.md new file mode 100644 index 0000000000..2559a7939f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/README.md @@ -0,0 +1,48 @@ +# Replace Text Failure - Placeholder Consumed After First Loop Iteration + +This scenario reproduces a `Replace Text` loop failure where the workflow +edits the template **in place** each iteration, so the first row consumes +the `[Name]` placeholder and every later row comes out unchanged. The job +completes **Successfully** with no error. + +## What this scenario uncovers + +**Root Cause:** The `For Each` loop's `Use Word File` scope points at the +template and saves it in place every iteration. Iteration 1 replaces +`[Name]` and persists it, so the placeholder no longer exists; iterations +2+ replace 0 occurrences. The per-iteration trace `Replaced 1` → `Replaced +0` → `Replaced 0` is the diagnostic tell. Distinct from +`replace-text-silent-no-substitution` (run-split, fails row 1 too) and +`replace-text-file-locked` (Auto Save `IOException`). + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-loop-template-overwrite.md` + +The correct agent behavior is to recognize a **Successful job with wrong +output on rows 2+** as a silent failure, tie it to in-place template editing +via the 1→0 logs, and recommend a fresh `Copy File` per iteration — not +dismiss the green job or blame a crash/lock. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `For Each` → `Use Word File` (`AutoSave`) on the template, `Replace Text in Document`, no Copy File | + +> **Note on fixtures.** No faulted job (State=Successful) and no error logs; +> the full job log carries the per-iteration `Replaced 1` → `Replaced 0` +> trace. The agent must broaden from a faulted-jobs query to the successful +> job and read the 1→0 pattern as the signal. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-loop-template-overwrite.md` +- Agent identified in-place template editing consuming the placeholder after + iteration 1 as the cause and recommended a fresh `Copy File` per iteration + (template read-only) — without treating the green job as healthy or + fabricating a crash/lock + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/1319741021.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/1319741021.json deleted file mode 100644 index 3579b35b1f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/1319741021.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T08:10:01.670Z", - "Message": "Execution started" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T08:10:03.910Z", - "Message": "[For Each recipient] Iteration 1 (Ava Stone): opened 'data\\OfferTemplate.docx'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T08:10:05.140Z", - "Message": "[Replace Text in Document] Iteration 1: Replaced 1 occurrence(s) of '[Name]'." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T08:10:07.520Z", - "Message": "[For Each recipient] Iteration 2 (Ben Cole): opened 'data\\OfferTemplate.docx'" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T08:10:08.730Z", - "Message": "[Replace Text in Document] Iteration 2: Replaced 0 occurrence(s) of '[Name]'." - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T08:10:11.050Z", - "Message": "[Replace Text in Document] Iteration 3: Replaced 0 occurrence(s) of '[Name]'." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T08:10:12.450Z", - "Message": "Execution ended in: 00:00:10" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/87d05f7b58.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/87d05f7b58.json deleted file mode 100644 index 89b9b504f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/87d05f7b58.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f", - "Name": "Bulk Letters", - "Path": "Bulk Letters", - "Description": "Production folder for bulk letter generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/README.md deleted file mode 100644 index 9a9497bb62..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Replace Text Failure - Placeholder Consumed After First Loop Iteration - -This scenario reproduces a `Replace Text` loop failure where the workflow -edits the template **in place** each iteration, so the first row consumes -the `[Name]` placeholder and every later row comes out unchanged. The job -completes **Successfully** with no error. - -## What this scenario uncovers - -**Root Cause:** The `For Each` loop's `Use Word File` scope points at the -template and saves it in place every iteration. Iteration 1 replaces -`[Name]` and persists it, so the placeholder no longer exists; iterations -2+ replace 0 occurrences. The per-iteration trace `Replaced 1` → `Replaced -0` → `Replaced 0` is the diagnostic tell. Distinct from -`replace-text-silent-no-substitution` (run-split, fails row 1 too) and -`replace-text-file-locked` (Auto Save `IOException`). - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-loop-template-overwrite.md` - -The correct agent behavior is to recognize a **Successful job with wrong -output on rows 2+** as a silent failure, tie it to in-place template editing -via the 1→0 logs, and recommend a fresh `Copy File` per iteration — not -dismiss the green job or blame a crash/lock. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `For Each` → `Use Word File` (`AutoSave`) on the template, `Replace Text in Document`, no Copy File | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** No faulted job (State=Successful) and no error logs; -> the full job log carries the per-iteration `Replaced 1` → `Replaced 0` -> trace. The agent must broaden from a faulted-jobs query to the successful -> job and read the 1→0 pattern as the signal. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-loop-template-overwrite.md` -- Agent identified in-place template editing consuming the placeholder after - iteration 1 as the cause and recommended a fresh `Copy File` per iteration - (template read-only) — without treating the green job as healthy or - fabricating a crash/lock diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/a4e082fcc7.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/a4e082fcc7.json deleted file mode 100644 index 09b75cfe54..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/a4e082fcc7.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70", - "StartTime": "2026-06-15T08:10:01.620Z", - "EndTime": "2026-06-15T08:10:12.480Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b2c3d4e5-f6a7-4315-a809-3c4d5e6f7081", - "Info": "The job completed successfully.", - "CreationTime": "2026-06-15T08:10:01.400Z", - "ReleaseName": "BulkOfferLetters", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-07", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021620, - "OrganizationUnitId": 7791618, - "FolderKey": "f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T08:10:12.480Z", - "JobError": null, - "ErrorCode": "", - "TraceId": "a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70", - "Id": 682452010 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/e74248ae3e.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/e74248ae3e.json deleted file mode 100644 index 8343806a7e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/e74248ae3e.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70", - "State": "Successful", - "ProcessName": "BulkOfferLetters", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-07", - "MachineKey": "", - "StartTime": "2026-06-15T08:10:01.620Z", - "EndTime": "2026-06-15T08:10:12.480Z", - "CreationTime": "2026-06-15T08:10:01.400Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/manifest.json deleted file mode 100644 index f56db2ca6f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Replace Text loop-overwrite scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "87d05f7b58.json" - }, - { - "match": "or jobs list --folder-key \"f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f\" --state Successful --output json", - "file": "e74248ae3e.json" - }, - { - "match": "or jobs list --folder-key f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f --state Successful --output json", - "file": "e74248ae3e.json" - }, - { - "match": "or jobs list --folder-key \"f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f\" --output json", - "file": "e74248ae3e.json" - }, - { - "match": "or jobs list --folder-key f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f --output json", - "file": "e74248ae3e.json" - }, - { - "match": "or jobs get \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\" --output json", - "file": "a4e082fcc7.json" - }, - { - "match": "or jobs get a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70 --output json", - "file": "a4e082fcc7.json" - }, - { - "match": "or jobs logs \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70 --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\" --output json", - "file": "1319741021.json" - }, - { - "match": "or jobs logs a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70 --output json", - "file": "1319741021.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/uip-fixture.json new file mode 100644 index 0000000000..99273aa22e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BulkOfferLetters\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-07\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T08:10:01.620Z\",\r\n \"EndTime\": \"2026-06-15T08:10:12.480Z\",\r\n \"CreationTime\": \"2026-06-15T08:10:01.400Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"BulkOfferLetters\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-07\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T08:10:01.620Z\",\r\n \"EndTime\": \"2026-06-15T08:10:12.480Z\",\r\n \"CreationTime\": \"2026-06-15T08:10:01.400Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\",\r\n \"StartTime\": \"2026-06-15T08:10:01.620Z\",\r\n \"EndTime\": \"2026-06-15T08:10:12.480Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b2c3d4e5-f6a7-4315-a809-3c4d5e6f7081\",\r\n \"Info\": \"The job completed successfully.\",\r\n \"CreationTime\": \"2026-06-15T08:10:01.400Z\",\r\n \"ReleaseName\": \"BulkOfferLetters\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-07\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021620,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T08:10:12.480Z\",\r\n \"JobError\": null,\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70\",\r\n \"Id\": 682452010\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-e5f6-4204-9708-2b3c4d5e6f70", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T08:10:01.670Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T08:10:03.910Z\",\r\n \"Message\": \"[For Each recipient] Iteration 1 (Ava Stone): opened 'data\\\\OfferTemplate.docx'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T08:10:05.140Z\",\r\n \"Message\": \"[Replace Text in Document] Iteration 1: Replaced 1 occurrence(s) of '[Name]'.\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T08:10:07.520Z\",\r\n \"Message\": \"[For Each recipient] Iteration 2 (Ben Cole): opened 'data\\\\OfferTemplate.docx'\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T08:10:08.730Z\",\r\n \"Message\": \"[Replace Text in Document] Iteration 2: Replaced 0 occurrence(s) of '[Name]'.\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T08:10:11.050Z\",\r\n \"Message\": \"[Replace Text in Document] Iteration 3: Replaced 0 occurrence(s) of '[Name]'.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T08:10:12.450Z\",\r\n \"Message\": \"Execution ended in: 00:00:10\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0a1b2c3-d4e5-4193-8607-1a2b3c4d5e6f\",\r\n \"Name\": \"Bulk Letters\",\r\n \"Path\": \"Bulk Letters\",\r\n \"Description\": \"Production folder for bulk letter generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/task.yaml index f217b90a53..ec36dd485a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-loop-overwrite/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/README.md new file mode 100644 index 0000000000..533b258c80 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/README.md @@ -0,0 +1,46 @@ +# Replace Text Failure - Multi-line Replacement Collapses / Loses Formatting + +This scenario reproduces a `Replace Text` formatting issue where a +multi-line address built with `Environment.NewLine` collapses to a single +line / loses formatting in the output. The job completes **Successfully**. + +## What this scenario uncovers + +**Root Cause:** The `Replace` value is `String.Join(Environment.NewLine, +...)`. `Replace Text` inserts the text as a flat run, so OS line breaks do +not become Word paragraph breaks — the address collapses to one line. The +match succeeds (1 occurrence), so the job is Successful; only the layout is +wrong. Not a template, Search-value, or package problem. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-multiline-formatting.md` + +The correct agent behavior is to recognize a **Successful job with wrong +layout** as a real issue, tie it to `Environment.NewLine` in the `Replace` +expression, and recommend Bookmarks / Form Fields + `Set Bookmark Text` (or +object-model paragraph breaks) — not blaming the template/package or +dismissing the green job. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Replace Text` with a `Replace` value built via `String.Join(Environment.NewLine, ...)` | + +> **Note on fixtures.** No faulted job (State=Successful) and no error logs; +> the trace shows the replacement succeeded (`Replaced 1 occurrence`). The +> agent diagnoses from the `Environment.NewLine` `Replace` expression in +> `Main.xaml` + the user's layout symptom — there is no error to grep. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-multiline-formatting.md` +- Agent identified raw `Environment.NewLine` in the `Replace` value as the + cause and recommended Bookmarks / Form Fields + `Set Bookmark Text` (or + object-model paragraph breaks; keeping `Replace Text` plain) — without + treating the green job as healthy or blaming the template/package + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/14a45eec9d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/14a45eec9d.json deleted file mode 100644 index a33264af64..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/14a45eec9d.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e5f6a7b8-c9d0-4204-9b1c-6f7081920314", - "State": "Successful", - "ProcessName": "AddressBlockFill", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-09", - "MachineKey": "", - "StartTime": "2026-06-15T10:48:01.250Z", - "EndTime": "2026-06-15T10:48:07.640Z", - "CreationTime": "2026-06-15T10:48:01.030Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/60027d23d5.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/60027d23d5.json deleted file mode 100644 index f6a69026fa..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/60027d23d5.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d4e5f6a7-b8c9-4193-8a0b-5e6f70819203", - "Name": "Address Blocks", - "Path": "Address Blocks", - "Description": "Production folder for address-block document generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/README.md deleted file mode 100644 index e54174efb9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/README.md +++ /dev/null @@ -1,45 +0,0 @@ -# Replace Text Failure - Multi-line Replacement Collapses / Loses Formatting - -This scenario reproduces a `Replace Text` formatting issue where a -multi-line address built with `Environment.NewLine` collapses to a single -line / loses formatting in the output. The job completes **Successfully**. - -## What this scenario uncovers - -**Root Cause:** The `Replace` value is `String.Join(Environment.NewLine, -...)`. `Replace Text` inserts the text as a flat run, so OS line breaks do -not become Word paragraph breaks — the address collapses to one line. The -match succeeds (1 occurrence), so the job is Successful; only the layout is -wrong. Not a template, Search-value, or package problem. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-multiline-formatting.md` - -The correct agent behavior is to recognize a **Successful job with wrong -layout** as a real issue, tie it to `Environment.NewLine` in the `Replace` -expression, and recommend Bookmarks / Form Fields + `Set Bookmark Text` (or -object-model paragraph breaks) — not blaming the template/package or -dismissing the green job. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Replace Text` with a `Replace` value built via `String.Join(Environment.NewLine, ...)` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** No faulted job (State=Successful) and no error logs; -> the trace shows the replacement succeeded (`Replaced 1 occurrence`). The -> agent diagnoses from the `Environment.NewLine` `Replace` expression in -> `Main.xaml` + the user's layout symptom — there is no error to grep. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-multiline-formatting.md` -- Agent identified raw `Environment.NewLine` in the `Replace` value as the - cause and recommended Bookmarks / Form Fields + `Set Bookmark Text` (or - object-model paragraph breaks; keeping `Replace Text` plain) — without - treating the green job as healthy or blaming the template/package diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/a529647073.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/a529647073.json deleted file mode 100644 index 20a037dbd4..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/a529647073.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e5f6a7b8-c9d0-4204-9b1c-6f7081920314", - "StartTime": "2026-06-15T10:48:01.250Z", - "EndTime": "2026-06-15T10:48:07.640Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f6a7b8c9-d0e1-4315-ac22-6f7081920314", - "Info": "The job completed successfully.", - "CreationTime": "2026-06-15T10:48:01.030Z", - "ReleaseName": "AddressBlockFill", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-09", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021840, - "OrganizationUnitId": 7791618, - "FolderKey": "d4e5f6a7-b8c9-4193-8a0b-5e6f70819203", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-15T10:48:07.640Z", - "JobError": null, - "ErrorCode": "", - "TraceId": "e5f6a7b8-c9d0-4204-9b1c-6f7081920314", - "Id": 682454332 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/e7231cf11e.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/e7231cf11e.json deleted file mode 100644 index 725464bc92..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/e7231cf11e.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-15T10:48:01.300Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T10:48:03.020Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T10:48:04.410Z", - "Message": "[Word Application Scope] Document 'data\\AddressTemplate.docx' opened" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-15T10:48:05.690Z", - "Message": "[Replace Text] Executed. Replaced 1 occurrence(s) of '[AddressBlock]'." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-15T10:48:07.610Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/manifest.json deleted file mode 100644 index f000aace22..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Replace Text multiline-formatting scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "60027d23d5.json" - }, - { - "match": "or jobs list --folder-key \"d4e5f6a7-b8c9-4193-8a0b-5e6f70819203\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key d4e5f6a7-b8c9-4193-8a0b-5e6f70819203 --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"d4e5f6a7-b8c9-4193-8a0b-5e6f70819203\" --state Successful --output json", - "file": "14a45eec9d.json" - }, - { - "match": "or jobs list --folder-key d4e5f6a7-b8c9-4193-8a0b-5e6f70819203 --state Successful --output json", - "file": "14a45eec9d.json" - }, - { - "match": "or jobs list --folder-key \"d4e5f6a7-b8c9-4193-8a0b-5e6f70819203\" --output json", - "file": "14a45eec9d.json" - }, - { - "match": "or jobs list --folder-key d4e5f6a7-b8c9-4193-8a0b-5e6f70819203 --output json", - "file": "14a45eec9d.json" - }, - { - "match": "or jobs get \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\" --output json", - "file": "a529647073.json" - }, - { - "match": "or jobs get e5f6a7b8-c9d0-4204-9b1c-6f7081920314 --output json", - "file": "a529647073.json" - }, - { - "match": "or jobs logs \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs e5f6a7b8-c9d0-4204-9b1c-6f7081920314 --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\" --output json", - "file": "e7231cf11e.json" - }, - { - "match": "or jobs logs e5f6a7b8-c9d0-4204-9b1c-6f7081920314 --output json", - "file": "e7231cf11e.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/uip-fixture.json new file mode 100644 index 0000000000..ae58c6f555 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d4e5f6a7-b8c9-4193-8a0b-5e6f70819203", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d4e5f6a7-b8c9-4193-8a0b-5e6f70819203", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"AddressBlockFill\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-09\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T10:48:01.250Z\",\r\n \"EndTime\": \"2026-06-15T10:48:07.640Z\",\r\n \"CreationTime\": \"2026-06-15T10:48:01.030Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e5f6a7b8-c9d0-4204-9b1c-6f7081920314", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d4e5f6a7-b8c9-4193-8a0b-5e6f70819203", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"AddressBlockFill\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-09\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-15T10:48:01.250Z\",\r\n \"EndTime\": \"2026-06-15T10:48:07.640Z\",\r\n \"CreationTime\": \"2026-06-15T10:48:01.030Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e5f6a7b8-c9d0-4204-9b1c-6f7081920314", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\",\r\n \"StartTime\": \"2026-06-15T10:48:01.250Z\",\r\n \"EndTime\": \"2026-06-15T10:48:07.640Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f6a7b8c9-d0e1-4315-ac22-6f7081920314\",\r\n \"Info\": \"The job completed successfully.\",\r\n \"CreationTime\": \"2026-06-15T10:48:01.030Z\",\r\n \"ReleaseName\": \"AddressBlockFill\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-09\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021840,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"d4e5f6a7-b8c9-4193-8a0b-5e6f70819203\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-15T10:48:07.640Z\",\r\n \"JobError\": null,\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"e5f6a7b8-c9d0-4204-9b1c-6f7081920314\",\r\n \"Id\": 682454332\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e5f6a7b8-c9d0-4204-9b1c-6f7081920314", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T10:48:01.300Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T10:48:03.020Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T10:48:04.410Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\AddressTemplate.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-15T10:48:05.690Z\",\r\n \"Message\": \"[Replace Text] Executed. Replaced 1 occurrence(s) of '[AddressBlock]'.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-15T10:48:07.610Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-4193-8a0b-5e6f70819203\",\r\n \"Name\": \"Address Blocks\",\r\n \"Path\": \"Address Blocks\",\r\n \"Description\": \"Production folder for address-block document generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/task.yaml index df8c560eee..720fc5f0d2 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-multiline-formatting/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/README.md new file mode 100644 index 0000000000..3b9087efea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/README.md @@ -0,0 +1,47 @@ +# Replace Text Failure - Silent No-Substitution (Run-Split Placeholder) + +This scenario reproduces a **silent** `Replace Text` failure: the job +completes **Successfully** but the `[Name]` placeholder is left unchanged in +the output, because the token is split across Word's internal XML runs in +the template and the exact-string search matches nothing. + +## What this scenario uncovers + +**Root Cause:** The `[Name]` placeholder in the template was edited in place, +so Word fragmented it across multiple `` runs. `Replace Text` searches +for the contiguous `[Name]`, matches zero occurrences, and replaces nothing +— the job reports success (matching nothing is not an error). The trace log +`Replaced 0 occurrence(s) of '[Name]'` is the diagnostic tell. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-silent-no-substitution.md` + +The correct agent behavior is to recognize a **Successful job with wrong +output** as a silent failure (not dismiss it because the job is green), tie +it to run-splitting via the "0 occurrences" log, and recommend retyping the +placeholder cleanly in one run or doing the substitution in code. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project; `Replace Text` searching for `[Name]` | + +> **Note on fixtures.** There are **no faulted jobs** and **no error logs** — +> the job is `Successful`. The agent must broaden from a faulted-jobs query +> to the successful job and read the `Replaced 0 occurrence(s) of '[Name]'` +> trace line as the signal. This tests whether the agent treats a green job +> with wrong output as a real failure. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-silent-no-substitution.md` +- Agent identified the run-split placeholder (exact-string search matching + nothing on a Successful job) as the cause and recommended retyping the + placeholder cleanly in one run and/or an in-code string/regex replace — + without treating the green job as healthy or fabricating a crash/lock + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/6b7dcb3858.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/6b7dcb3858.json deleted file mode 100644 index 13a5665c13..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/6b7dcb3858.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "c4d5e6f7-a8b9-4193-0c1d-2e3f40516273", - "Name": "Offer Docs", - "Path": "Offer Docs", - "Description": "Production folder for onboarding document generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/README.md deleted file mode 100644 index 2f57bc69d2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Replace Text Failure - Silent No-Substitution (Run-Split Placeholder) - -This scenario reproduces a **silent** `Replace Text` failure: the job -completes **Successfully** but the `[Name]` placeholder is left unchanged in -the output, because the token is split across Word's internal XML runs in -the template and the exact-string search matches nothing. - -## What this scenario uncovers - -**Root Cause:** The `[Name]` placeholder in the template was edited in place, -so Word fragmented it across multiple `` runs. `Replace Text` searches -for the contiguous `[Name]`, matches zero occurrences, and replaces nothing -— the job reports success (matching nothing is not an error). The trace log -`Replaced 0 occurrence(s) of '[Name]'` is the diagnostic tell. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-silent-no-substitution.md` - -The correct agent behavior is to recognize a **Successful job with wrong -output** as a silent failure (not dismiss it because the job is green), tie -it to run-splitting via the "0 occurrences" log, and recommend retyping the -placeholder cleanly in one run or doing the substitution in code. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project; `Replace Text` searching for `[Name]` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** There are **no faulted jobs** and **no error logs** — -> the job is `Successful`. The agent must broaden from a faulted-jobs query -> to the successful job and read the `Replaced 0 occurrence(s) of '[Name]'` -> trace line as the signal. This tests whether the agent treats a green job -> with wrong output as a real failure. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-silent-no-substitution.md` -- Agent identified the run-split placeholder (exact-string search matching - nothing on a Successful job) as the cause and recommended retyping the - placeholder cleanly in one run and/or an in-code string/regex replace — - without treating the green job as healthy or fabricating a crash/lock diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/c7717a7a9d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/c7717a7a9d.json deleted file mode 100644 index ecc1dab484..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/c7717a7a9d.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-14T09:02:01.560Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T09:02:03.180Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T09:02:04.420Z", - "Message": "[Word Application Scope] Document 'data\\OfferTemplate.docx' opened" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-14T09:02:05.610Z", - "Message": "[Replace Text] Executed. Replaced 0 occurrence(s) of '[Name]'." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-14T09:02:06.910Z", - "Message": "Execution ended in: 00:00:05" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/ceb4a83764.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/ceb4a83764.json deleted file mode 100644 index 03626abdb0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/ceb4a83764.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d5e6f7a8-b9c0-4204-1d2e-3f4051627384", - "StartTime": "2026-06-14T09:02:01.510Z", - "EndTime": "2026-06-14T09:02:06.940Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "e6f7a8b9-c0d1-4315-1d2e-3f4051627395", - "Info": "The job completed successfully.", - "CreationTime": "2026-06-14T09:02:01.280Z", - "ReleaseName": "OnboardingLetter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-05", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6021342, - "OrganizationUnitId": 7791618, - "FolderKey": "c4d5e6f7-a8b9-4193-0c1d-2e3f40516273", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-14T09:02:06.940Z", - "JobError": null, - "ErrorCode": "", - "TraceId": "d5e6f7a8-b9c0-4204-1d2e-3f4051627384", - "Id": 682447702 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/f3cf204185.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/f3cf204185.json deleted file mode 100644 index 9b7537bce7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/f3cf204185.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d5e6f7a8-b9c0-4204-1d2e-3f4051627384", - "State": "Successful", - "ProcessName": "OnboardingLetter", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-05", - "MachineKey": "", - "StartTime": "2026-06-14T09:02:01.510Z", - "EndTime": "2026-06-14T09:02:06.940Z", - "CreationTime": "2026-06-14T09:02:01.280Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/manifest.json deleted file mode 100644 index a870b8902e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Replace Text silent-no-substitution scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "6b7dcb3858.json" - }, - { - "match": "or jobs list --folder-key \"c4d5e6f7-a8b9-4193-0c1d-2e3f40516273\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key c4d5e6f7-a8b9-4193-0c1d-2e3f40516273 --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"c4d5e6f7-a8b9-4193-0c1d-2e3f40516273\" --state Successful --output json", - "file": "f3cf204185.json" - }, - { - "match": "or jobs list --folder-key c4d5e6f7-a8b9-4193-0c1d-2e3f40516273 --state Successful --output json", - "file": "f3cf204185.json" - }, - { - "match": "or jobs list --folder-key \"c4d5e6f7-a8b9-4193-0c1d-2e3f40516273\" --output json", - "file": "f3cf204185.json" - }, - { - "match": "or jobs list --folder-key c4d5e6f7-a8b9-4193-0c1d-2e3f40516273 --output json", - "file": "f3cf204185.json" - }, - { - "match": "or jobs get \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\" --output json", - "file": "ceb4a83764.json" - }, - { - "match": "or jobs get d5e6f7a8-b9c0-4204-1d2e-3f4051627384 --output json", - "file": "ceb4a83764.json" - }, - { - "match": "or jobs logs \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs d5e6f7a8-b9c0-4204-1d2e-3f4051627384 --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\" --output json", - "file": "c7717a7a9d.json" - }, - { - "match": "or jobs logs d5e6f7a8-b9c0-4204-1d2e-3f4051627384 --output json", - "file": "c7717a7a9d.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/uip-fixture.json new file mode 100644 index 0000000000..12f4dc2e2d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c4d5e6f7-a8b9-4193-0c1d-2e3f40516273", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c4d5e6f7-a8b9-4193-0c1d-2e3f40516273", + "--state", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"OnboardingLetter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-05\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-14T09:02:01.510Z\",\r\n \"EndTime\": \"2026-06-14T09:02:06.940Z\",\r\n \"CreationTime\": \"2026-06-14T09:02:01.280Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d5e6f7a8-b9c0-4204-1d2e-3f4051627384", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "c4d5e6f7-a8b9-4193-0c1d-2e3f40516273", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"OnboardingLetter\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-05\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-14T09:02:01.510Z\",\r\n \"EndTime\": \"2026-06-14T09:02:06.940Z\",\r\n \"CreationTime\": \"2026-06-14T09:02:01.280Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d5e6f7a8-b9c0-4204-1d2e-3f4051627384", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\",\r\n \"StartTime\": \"2026-06-14T09:02:01.510Z\",\r\n \"EndTime\": \"2026-06-14T09:02:06.940Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"e6f7a8b9-c0d1-4315-1d2e-3f4051627395\",\r\n \"Info\": \"The job completed successfully.\",\r\n \"CreationTime\": \"2026-06-14T09:02:01.280Z\",\r\n \"ReleaseName\": \"OnboardingLetter\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-05\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6021342,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"c4d5e6f7-a8b9-4193-0c1d-2e3f40516273\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-14T09:02:06.940Z\",\r\n \"JobError\": null,\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"d5e6f7a8-b9c0-4204-1d2e-3f4051627384\",\r\n \"Id\": 682447702\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d5e6f7a8-b9c0-4204-1d2e-3f4051627384", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T09:02:01.560Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T09:02:03.180Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T09:02:04.420Z\",\r\n \"Message\": \"[Word Application Scope] Document 'data\\\\OfferTemplate.docx' opened\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-14T09:02:05.610Z\",\r\n \"Message\": \"[Replace Text] Executed. Replaced 0 occurrence(s) of '[Name]'.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-14T09:02:06.910Z\",\r\n \"Message\": \"Execution ended in: 00:00:05\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c4d5e6f7-a8b9-4193-0c1d-2e3f40516273\",\r\n \"Name\": \"Offer Docs\",\r\n \"Path\": \"Offer Docs\",\r\n \"Description\": \"Production folder for onboarding document generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/task.yaml index 11804c63b5..a67823f793 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-silent-no-substitution/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/README.md new file mode 100644 index 0000000000..d17bde30ff --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/README.md @@ -0,0 +1,52 @@ +# Replace Text Failure - TargetInvocationException / Studio Crash on Drop + +This scenario reproduces a **design-time** Studio crash: dropping the +`Replace Text` activity (or opening the workflow) throws +`System.Reflection.TargetInvocationException` because the project pins a +`UiPath.Word.Activities` version newer than the installed Studio (2021.10) +can construct. The process has **never run** — there are no Orchestrator +jobs. + +## What this scenario uncovers + +**Root Cause:** `project.json` pins `UiPath.Word.Activities [2.0.0]` against +`studioVersion 21.10.5.0`. Studio 2021.10 cannot construct the newer +package's activity designer, so the reflection call crashes at design time. +The diagnosis comes from the **project source + the stated Studio version**, +not job evidence. + +This maps to: +`references/activity-packages/word-activities/playbooks/replace-text-version-mismatch.md` + +The correct agent behavior is to reason from `project.json` (package vs +Studio version) rather than chasing Orchestrator jobs, match the +version-mismatch playbook, and recommend aligning versions via Manage +Packages — explicitly NOT the runtime `Cannot create unknown type` package +restore fix. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project pinning `UiPath.Word.Activities [2.0.0]` with `studioVersion 21.10.5.0` | + +> **Note on fixtures.** This is the one Word scenario with **no faulted +> job** by design — it is a design-time Studio crash, so `or jobs list` +> returns empty and the agent must diagnose from `project.json` + the +> Studio version in the prompt. It tests whether the agent correctly +> distinguishes a design-time version mismatch from the runtime +> `cannot-create-unknown-type` package gap. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `replace-text-version-mismatch.md` +- Agent identified a Studio↔`UiPath.Word.Activities` version mismatch (newer + package vs older Studio 2021.10) as the design-time cause, reasoning from + the project source, and recommended aligning versions via Manage Packages + (downgrade the package or upgrade Studio) — without confusing it with the + runtime `cannot-create-unknown-type` fault or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/52a6732454.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/52a6732454.json deleted file mode 100644 index 83729d255e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/52a6732454.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a8b9c0d1-e2f3-4193-4051-627384950617", - "Name": "Doc Dev", - "Path": "Doc Dev", - "Description": "Development folder for document automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/README.md deleted file mode 100644 index ad70475eda..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/README.md +++ /dev/null @@ -1,51 +0,0 @@ -# Replace Text Failure - TargetInvocationException / Studio Crash on Drop - -This scenario reproduces a **design-time** Studio crash: dropping the -`Replace Text` activity (or opening the workflow) throws -`System.Reflection.TargetInvocationException` because the project pins a -`UiPath.Word.Activities` version newer than the installed Studio (2021.10) -can construct. The process has **never run** — there are no Orchestrator -jobs. - -## What this scenario uncovers - -**Root Cause:** `project.json` pins `UiPath.Word.Activities [2.0.0]` against -`studioVersion 21.10.5.0`. Studio 2021.10 cannot construct the newer -package's activity designer, so the reflection call crashes at design time. -The diagnosis comes from the **project source + the stated Studio version**, -not job evidence. - -This maps to: -`references/activity-packages/word-activities/playbooks/replace-text-version-mismatch.md` - -The correct agent behavior is to reason from `project.json` (package vs -Studio version) rather than chasing Orchestrator jobs, match the -version-mismatch playbook, and recommend aligning versions via Manage -Packages — explicitly NOT the runtime `Cannot create unknown type` package -restore fix. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project pinning `UiPath.Word.Activities [2.0.0]` with `studioVersion 21.10.5.0` | -| `fixtures/mocks/responses/*.json` | folders list + **empty** job lists (the process never ran) | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** This is the one Word scenario with **no faulted -> job** by design — it is a design-time Studio crash, so `or jobs list` -> returns empty and the agent must diagnose from `project.json` + the -> Studio version in the prompt. It tests whether the agent correctly -> distinguishes a design-time version mismatch from the runtime -> `cannot-create-unknown-type` package gap. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `replace-text-version-mismatch.md` -- Agent identified a Studio↔`UiPath.Word.Activities` version mismatch (newer - package vs older Studio 2021.10) as the design-time cause, reasoning from - the project source, and recommended aligning versions via Manage Packages - (downgrade the package or upgrade Studio) — without confusing it with the - runtime `cannot-create-unknown-type` fault or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/manifest.json deleted file mode 100644 index 7458cc5850..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Replace Text version-mismatch scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "52a6732454.json" - }, - { - "match": "or jobs list --folder-key \"a8b9c0d1-e2f3-4193-4051-627384950617\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key a8b9c0d1-e2f3-4193-4051-627384950617 --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"a8b9c0d1-e2f3-4193-4051-627384950617\" --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key a8b9c0d1-e2f3-4193-4051-627384950617 --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands (no jobs exist for this design-time crash)." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..4e45572d01 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/data/uip-fixture.json @@ -0,0 +1,51 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a8b9c0d1-e2f3-4193-4051-627384950617", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a8b9c0d1-e2f3-4193-4051-627384950617", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a8b9c0d1-e2f3-4193-4051-627384950617\",\r\n \"Name\": \"Doc Dev\",\r\n \"Path\": \"Doc Dev\",\r\n \"Description\": \"Development folder for document automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/task.yaml index 3d42df9134..e16ab32167 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-replace-text-version-mismatch/task.yaml @@ -24,22 +24,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/README.md new file mode 100644 index 0000000000..8dcb02e243 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/README.md @@ -0,0 +1,49 @@ +# Word Application Scope Failure - "The file appears to be corrupted" + +This scenario reproduces a `Word Application Scope` failure where the +document opens reporting `The file appears to be corrupted`. The workflow +edits a template **in place** and an orphaned WINWORD.EXE from a prior run +holds the file lock, leaving the template half-written. + +## What this scenario uncovers + +**Root Cause:** The workflow opens `MonthlyTemplate.docx` and edits it in +place (no save-as-new). A Word instance was already running on the host +when the scope started (orphaned WINWORD.EXE), so the file is locked / +half-written and Word reports it as corrupted. "Worked the first few runs, +opens fine on my own machine" points at a host-state / locking problem - +not a workflow-logic defect or a genuinely corrupt template. + +This maps to: +`references/activity-packages/word-activities/playbooks/word-scope-file-corrupted.md` + +The correct agent behavior is to tie the corruption error to an +orphaned/locked Word session plus the in-place template overwrite, and +recommend clearing the lock (Kill Process WINWORD before the scope) and +saving to a new file - not to declare the template irrecoverably corrupt or +attempt host commands itself (the user is off-host). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project whose `Word Application Scope` edits a template in place | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. The full job log includes a +> "Word already running on the host" warning that points at the orphaned +> WINWORD.EXE lock. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `word-scope-file-corrupted.md` +- Agent identified an orphaned/locked Word session and/or in-place template + overwrite as the cause and recommended clearing the lock (Kill Process + WINWORD before the scope) and/or saving to a new file - without + fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/1a3c1022f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/1a3c1022f5.json deleted file mode 100644 index a9d8e746f0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/1a3c1022f5.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-10T06:00:05.360Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-10T06:00:07.120Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Warn", - "TimeStamp": "2026-06-10T06:00:08.640Z", - "Message": "[Word Application Scope] A Microsoft Word instance was already running on the host when the scope started." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T06:00:11.790Z", - "Message": "[Word Application Scope] Word experienced an error trying to open the file 'C:\\UiPath\\Reporting\\data\\MonthlyTemplate.docx'. The file appears to be corrupted." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T06:00:11.840Z", - "Message": "Word experienced an error trying to open the file. The file appears to be corrupted.\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-10T06:00:11.860Z", - "Message": "Execution ended in: 00:00:06" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/1d70fcb46f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/1d70fcb46f.json deleted file mode 100644 index cfe5309fdc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/1d70fcb46f.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403", - "Name": "Reporting", - "Path": "Reporting", - "Description": "Production folder for scheduled reporting processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/8b7c6bb54d.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/8b7c6bb54d.json deleted file mode 100644 index bb84605b02..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/8b7c6bb54d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514", - "StartTime": "2026-06-10T06:00:05.310Z", - "EndTime": "2026-06-10T06:00:11.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "f7a8b9c0-d1e2-4315-c7d8-e9f0a1253647", - "Info": "Word Application Scope: Word experienced an error trying to open the file 'C:\\UiPath\\Reporting\\data\\MonthlyTemplate.docx'. The file appears to be corrupted.\r\n\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.Word.Activities.WordException: Word experienced an error trying to open the file. The file appears to be corrupted.", - "CreationTime": "2026-06-10T06:00:05.020Z", - "ReleaseName": "MonthlyReportFiller", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020655, - "OrganizationUnitId": 7791618, - "FolderKey": "d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-10T06:00:11.870Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514", - "Id": 682419901 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/README.md deleted file mode 100644 index 9911b5fe00..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Word Application Scope Failure - "The file appears to be corrupted" - -This scenario reproduces a `Word Application Scope` failure where the -document opens reporting `The file appears to be corrupted`. The workflow -edits a template **in place** and an orphaned WINWORD.EXE from a prior run -holds the file lock, leaving the template half-written. - -## What this scenario uncovers - -**Root Cause:** The workflow opens `MonthlyTemplate.docx` and edits it in -place (no save-as-new). A Word instance was already running on the host -when the scope started (orphaned WINWORD.EXE), so the file is locked / -half-written and Word reports it as corrupted. "Worked the first few runs, -opens fine on my own machine" points at a host-state / locking problem - -not a workflow-logic defect or a genuinely corrupt template. - -This maps to: -`references/activity-packages/word-activities/playbooks/word-scope-file-corrupted.md` - -The correct agent behavior is to tie the corruption error to an -orphaned/locked Word session plus the in-place template overwrite, and -recommend clearing the lock (Kill Process WINWORD before the scope) and -saving to a new file - not to declare the template irrecoverably corrupt or -attempt host commands itself (the user is off-host). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project whose `Word Application Scope` edits a template in place | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. The full job log includes a -> "Word already running on the host" warning that points at the orphaned -> WINWORD.EXE lock. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `word-scope-file-corrupted.md` -- Agent identified an orphaned/locked Word session and/or in-place template - overwrite as the cause and recommended clearing the lock (Kill Process - WINWORD before the scope) and/or saving to a new file - without - fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/d433e4f05a.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/d433e4f05a.json deleted file mode 100644 index 9eafc4cfe1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/d433e4f05a.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-10T06:00:11.790Z", - "Message": "[Word Application Scope] Word experienced an error trying to open the file 'C:\\UiPath\\Reporting\\data\\MonthlyTemplate.docx'. The file appears to be corrupted." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-10T06:00:11.840Z", - "Message": "Word experienced an error trying to open the file. The file appears to be corrupted.\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/dd0c8c19a2.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/dd0c8c19a2.json deleted file mode 100644 index f28462812b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/dd0c8c19a2.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514", - "State": "Faulted", - "ProcessName": "MonthlyReportFiller", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-10T06:00:05.310Z", - "EndTime": "2026-06-10T06:00:11.870Z", - "CreationTime": "2026-06-10T06:00:05.020Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/manifest.json deleted file mode 100644 index 81ca19eff2..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Application Scope file-corrupted scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "1d70fcb46f.json" - }, - { - "match": "or jobs list --folder-key \"d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403\" --state Faulted --output json", - "file": "dd0c8c19a2.json" - }, - { - "match": "or jobs list --folder-key d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403 --state Faulted --output json", - "file": "dd0c8c19a2.json" - }, - { - "match": "or jobs get \"e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514\" --output json", - "file": "8b7c6bb54d.json" - }, - { - "match": "or jobs get e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514 --output json", - "file": "8b7c6bb54d.json" - }, - { - "match": "or jobs logs \"e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514\" --level Error --output json", - "file": "d433e4f05a.json" - }, - { - "match": "or jobs logs e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514 --level Error --output json", - "file": "d433e4f05a.json" - }, - { - "match": "or jobs logs \"e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514\" --output json", - "file": "1a3c1022f5.json" - }, - { - "match": "or jobs logs e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514 --output json", - "file": "1a3c1022f5.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/uip-fixture.json new file mode 100644 index 0000000000..f02830d8fb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MonthlyReportFiller\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-10T06:00:05.310Z\",\r\n \"EndTime\": \"2026-06-10T06:00:11.870Z\",\r\n \"CreationTime\": \"2026-06-10T06:00:05.020Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T06:00:11.790Z\",\r\n \"Message\": \"[Word Application Scope] Word experienced an error trying to open the file 'C:\\\\UiPath\\\\Reporting\\\\data\\\\MonthlyTemplate.docx'. The file appears to be corrupted.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T06:00:11.840Z\",\r\n \"Message\": \"Word experienced an error trying to open the file. The file appears to be corrupted.\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514\",\r\n \"StartTime\": \"2026-06-10T06:00:05.310Z\",\r\n \"EndTime\": \"2026-06-10T06:00:11.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"f7a8b9c0-d1e2-4315-c7d8-e9f0a1253647\",\r\n \"Info\": \"Word Application Scope: Word experienced an error trying to open the file 'C:\\\\UiPath\\\\Reporting\\\\data\\\\MonthlyTemplate.docx'. The file appears to be corrupted.\\r\\n\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.Word.Activities.WordException: Word experienced an error trying to open the file. The file appears to be corrupted.\",\r\n \"CreationTime\": \"2026-06-10T06:00:05.020Z\",\r\n \"ReleaseName\": \"MonthlyReportFiller\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020655,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-10T06:00:11.870Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514\",\r\n \"Id\": 682419901\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e6f7a8b9-c0d1-4204-b6c7-d8e9f0a12514", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T06:00:05.360Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T06:00:07.120Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Warn\",\r\n \"TimeStamp\": \"2026-06-10T06:00:08.640Z\",\r\n \"Message\": \"[Word Application Scope] A Microsoft Word instance was already running on the host when the scope started.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T06:00:11.790Z\",\r\n \"Message\": \"[Word Application Scope] Word experienced an error trying to open the file 'C:\\\\UiPath\\\\Reporting\\\\data\\\\MonthlyTemplate.docx'. The file appears to be corrupted.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-10T06:00:11.840Z\",\r\n \"Message\": \"Word experienced an error trying to open the file. The file appears to be corrupted.\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-10T06:00:11.860Z\",\r\n \"Message\": \"Execution ended in: 00:00:06\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4e5f6a7-b8c9-4193-a5b6-c7d8e9f01403\",\r\n \"Name\": \"Reporting\",\r\n \"Path\": \"Reporting\",\r\n \"Description\": \"Production folder for scheduled reporting processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/task.yaml index 6ccf3b988c..6a6ee30d86 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-corrupted/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/README.md new file mode 100644 index 0000000000..156a1ec4a2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/README.md @@ -0,0 +1,48 @@ +# Word Application Scope Failure - File Path Not Found (Relative Path) + +This scenario reproduces a `Word Application Scope` `FileNotFoundException` +caused by a **relative document path** that resolves against the unattended +robot's working directory instead of the project folder. + +## What this scenario uncovers + +**Root Cause:** The `Word Application Scope`'s `FileName` is the relative +path `Output\OfferLetter.docx`. On the robot, the relative path resolves +against `C:\Windows\System32\config\systemprofile` (the service account +profile / robot working directory), not the project folder, so the file is +not found. "Works from Studio, fails on the robot" confirms a +working-directory resolution gap, not a missing file or a Word defect. The +resolved path in the error message is the tell. + +This maps to: +`references/activity-packages/word-activities/playbooks/word-scope-file-path-not-found.md` + +The correct agent behavior is to tie the resolved path to relative-path +resolution and recommend an absolute path / `Path.Combine` (or +`Create if not exists` when the document is generated) - NOT to edit the +document content or declare the file genuinely missing. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project whose `Word Application Scope` uses a relative `FileName` | + +> **Note on fixtures.** The faulted job's error resolves the relative path +> to `C:\Windows\System32\config\systemprofile\Output\OfferLetter.docx` - +> the service-profile root an unattended robot uses as its working +> directory - which is the diagnostic signal distinguishing this from a +> genuinely missing file. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `word-scope-file-path-not-found.md` +- Agent identified the relative-path-resolving-against-the-robot-working- + directory cause and recommended an absolute path / `Path.Combine` (or + `Create if not exists` when the document is generated) - without editing + document content or fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/267fbf18ff.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/267fbf18ff.json deleted file mode 100644 index 8e16877514..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/267fbf18ff.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c9d0e1f2-a3b4-4204-d7e8-f90124354617", - "State": "Faulted", - "ProcessName": "OfferLetterGen", - "ReleaseKey": "", - "Source": "Queue", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-03", - "MachineKey": "", - "StartTime": "2026-06-12T08:20:01.540Z", - "EndTime": "2026-06-12T08:20:03.220Z", - "CreationTime": "2026-06-12T08:20:01.300Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/3294f708cb.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/3294f708cb.json deleted file mode 100644 index 829883881b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/3294f708cb.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b8c9d0e1-f2a3-4193-c6d7-e8f901243506", - "Name": "Offer Letters", - "Path": "Offer Letters", - "Description": "Production folder for HR offer-letter generation.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/README.md deleted file mode 100644 index b2c61c8f38..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Word Application Scope Failure - File Path Not Found (Relative Path) - -This scenario reproduces a `Word Application Scope` `FileNotFoundException` -caused by a **relative document path** that resolves against the unattended -robot's working directory instead of the project folder. - -## What this scenario uncovers - -**Root Cause:** The `Word Application Scope`'s `FileName` is the relative -path `Output\OfferLetter.docx`. On the robot, the relative path resolves -against `C:\Windows\System32\config\systemprofile` (the service account -profile / robot working directory), not the project folder, so the file is -not found. "Works from Studio, fails on the robot" confirms a -working-directory resolution gap, not a missing file or a Word defect. The -resolved path in the error message is the tell. - -This maps to: -`references/activity-packages/word-activities/playbooks/word-scope-file-path-not-found.md` - -The correct agent behavior is to tie the resolved path to relative-path -resolution and recommend an absolute path / `Path.Combine` (or -`Create if not exists` when the document is generated) - NOT to edit the -document content or declare the file genuinely missing. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project whose `Word Application Scope` uses a relative `FileName` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** The faulted job's error resolves the relative path -> to `C:\Windows\System32\config\systemprofile\Output\OfferLetter.docx` - -> the service-profile root an unattended robot uses as its working -> directory - which is the diagnostic signal distinguishing this from a -> genuinely missing file. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `word-scope-file-path-not-found.md` -- Agent identified the relative-path-resolving-against-the-robot-working- - directory cause and recommended an absolute path / `Path.Combine` (or - `Create if not exists` when the document is generated) - without editing - document content or fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/b7d5567ba1.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/b7d5567ba1.json deleted file mode 100644 index 8914a38fcb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/b7d5567ba1.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c9d0e1f2-a3b4-4204-d7e8-f90124354617", - "StartTime": "2026-06-12T08:20:01.540Z", - "EndTime": "2026-06-12T08:20:03.220Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Queue", - "SourceType": "Queue", - "BatchExecutionKey": "d0e1f2a3-b4c5-4315-d7e8-f90124354628", - "Info": "Word Application Scope: Could not find file 'C:\\Windows\\System32\\config\\systemprofile\\Output\\OfferLetter.docx'.\r\n\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.FileNotFoundException: Could not find file 'C:\\Windows\\System32\\config\\systemprofile\\Output\\OfferLetter.docx'.", - "CreationTime": "2026-06-12T08:20:01.300Z", - "ReleaseName": "OfferLetterGen", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-03", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020901, - "OrganizationUnitId": 7791618, - "FolderKey": "b8c9d0e1-f2a3-4193-c6d7-e8f901243506", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "scheduler", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-12T08:20:03.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c9d0e1f2-a3b4-4204-d7e8-f90124354617", - "Id": 682431204 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/c68cd41d51.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/c68cd41d51.json deleted file mode 100644 index 3a59cc1e3e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/c68cd41d51.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-12T08:20:01.590Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-12T08:20:02.870Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-12T08:20:03.150Z", - "Message": "[Word Application Scope] Could not find file 'C:\\Windows\\System32\\config\\systemprofile\\Output\\OfferLetter.docx'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-12T08:20:03.190Z", - "Message": "Could not find file 'C:\\Windows\\System32\\config\\systemprofile\\Output\\OfferLetter.docx'.\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-12T08:20:03.210Z", - "Message": "Execution ended in: 00:00:01" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/eb7c9f47a2.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/eb7c9f47a2.json deleted file mode 100644 index b37cdeaeea..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/eb7c9f47a2.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-12T08:20:03.150Z", - "Message": "[Word Application Scope] Could not find file 'C:\\Windows\\System32\\config\\systemprofile\\Output\\OfferLetter.docx'." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-12T08:20:03.190Z", - "Message": "Could not find file 'C:\\Windows\\System32\\config\\systemprofile\\Output\\OfferLetter.docx'.\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/manifest.json deleted file mode 100644 index c641e0dc4d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Application Scope file-path scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "3294f708cb.json" - }, - { - "match": "or jobs list --folder-key \"b8c9d0e1-f2a3-4193-c6d7-e8f901243506\" --state Faulted --output json", - "file": "267fbf18ff.json" - }, - { - "match": "or jobs list --folder-key b8c9d0e1-f2a3-4193-c6d7-e8f901243506 --state Faulted --output json", - "file": "267fbf18ff.json" - }, - { - "match": "or jobs get \"c9d0e1f2-a3b4-4204-d7e8-f90124354617\" --output json", - "file": "b7d5567ba1.json" - }, - { - "match": "or jobs get c9d0e1f2-a3b4-4204-d7e8-f90124354617 --output json", - "file": "b7d5567ba1.json" - }, - { - "match": "or jobs logs \"c9d0e1f2-a3b4-4204-d7e8-f90124354617\" --level Error --output json", - "file": "eb7c9f47a2.json" - }, - { - "match": "or jobs logs c9d0e1f2-a3b4-4204-d7e8-f90124354617 --level Error --output json", - "file": "eb7c9f47a2.json" - }, - { - "match": "or jobs logs \"c9d0e1f2-a3b4-4204-d7e8-f90124354617\" --output json", - "file": "c68cd41d51.json" - }, - { - "match": "or jobs logs c9d0e1f2-a3b4-4204-d7e8-f90124354617 --output json", - "file": "c68cd41d51.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/uip-fixture.json new file mode 100644 index 0000000000..0da2ee1979 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b8c9d0e1-f2a3-4193-c6d7-e8f901243506", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c9d0e1f2-a3b4-4204-d7e8-f90124354617\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OfferLetterGen\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Queue\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-03\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-12T08:20:01.540Z\",\r\n \"EndTime\": \"2026-06-12T08:20:03.220Z\",\r\n \"CreationTime\": \"2026-06-12T08:20:01.300Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c9d0e1f2-a3b4-4204-d7e8-f90124354617", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T08:20:03.150Z\",\r\n \"Message\": \"[Word Application Scope] Could not find file 'C:\\\\Windows\\\\System32\\\\config\\\\systemprofile\\\\Output\\\\OfferLetter.docx'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T08:20:03.190Z\",\r\n \"Message\": \"Could not find file 'C:\\\\Windows\\\\System32\\\\config\\\\systemprofile\\\\Output\\\\OfferLetter.docx'.\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c9d0e1f2-a3b4-4204-d7e8-f90124354617", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c9d0e1f2-a3b4-4204-d7e8-f90124354617\",\r\n \"StartTime\": \"2026-06-12T08:20:01.540Z\",\r\n \"EndTime\": \"2026-06-12T08:20:03.220Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"BatchExecutionKey\": \"d0e1f2a3-b4c5-4315-d7e8-f90124354628\",\r\n \"Info\": \"Word Application Scope: Could not find file 'C:\\\\Windows\\\\System32\\\\config\\\\systemprofile\\\\Output\\\\OfferLetter.docx'.\\r\\n\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.FileNotFoundException: Could not find file 'C:\\\\Windows\\\\System32\\\\config\\\\systemprofile\\\\Output\\\\OfferLetter.docx'.\",\r\n \"CreationTime\": \"2026-06-12T08:20:01.300Z\",\r\n \"ReleaseName\": \"OfferLetterGen\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-03\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020901,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b8c9d0e1-f2a3-4193-c6d7-e8f901243506\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"scheduler\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-12T08:20:03.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c9d0e1f2-a3b4-4204-d7e8-f90124354617\",\r\n \"Id\": 682431204\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c9d0e1f2-a3b4-4204-d7e8-f90124354617", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-12T08:20:01.590Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-12T08:20:02.870Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T08:20:03.150Z\",\r\n \"Message\": \"[Word Application Scope] Could not find file 'C:\\\\Windows\\\\System32\\\\config\\\\systemprofile\\\\Output\\\\OfferLetter.docx'.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-12T08:20:03.190Z\",\r\n \"Message\": \"Could not find file 'C:\\\\Windows\\\\System32\\\\config\\\\systemprofile\\\\Output\\\\OfferLetter.docx'.\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-12T08:20:03.210Z\",\r\n \"Message\": \"Execution ended in: 00:00:01\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b8c9d0e1-f2a3-4193-c6d7-e8f901243506\",\r\n \"Name\": \"Offer Letters\",\r\n \"Path\": \"Offer Letters\",\r\n \"Description\": \"Production folder for HR offer-letter generation.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/task.yaml index e7570be53b..474b396c7a 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-file-path/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/README.md new file mode 100644 index 0000000000..1b88ee5e80 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/README.md @@ -0,0 +1,49 @@ +# Word Application Scope Failure - Workflow Hangs on a Background Dialog + +This scenario reproduces a `Word Application Scope` **hang**: the scope +opens a password-protected document, Word raises a password prompt the +unattended robot cannot answer, and the modal dialog blocks all COM calls. +The job never advances past the scope and is cancelled after ~30 minutes +with no error. + +## What this scenario uncovers + +**Root Cause:** The `Word Application Scope` opens `NDA-Protected.docx` +with no `Password` and `Visible=False`. Word's background password prompt +(invisible unattended) wedges the COM calls, so the scope hangs. The tell +is the absence of any error: a blocking dialog produces a hang, not an +exception, and the logs stop at "Opening document". + +This maps to: +`references/activity-packages/word-activities/playbooks/word-scope-hangs-background-prompt.md` + +The correct agent behavior is to recognize the no-error hang as a hidden +modal dialog (most likely the password prompt for the protected document) +and recommend reproducing with Word visible, then supplying the password / +clearing the prompt - not to attempt host commands itself (the user is +off-host). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project whose `Word Application Scope` opens a protected doc with no password, Visible=False | + +> **Note on fixtures.** There are no faulted jobs - the hung job shows as +> **Stopped** (cancelled), and `or jobs logs --level Error` is empty. The +> full log ends at "Opening document 'data\\NDA-Protected.docx'" and jumps +> to "Execution was canceled". The agent must broaden from a faulted-jobs +> query to find the stopped job and read the no-error hang as the signal. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `word-scope-hangs-background-prompt.md` +- Agent identified a hidden background Word dialog (password prompt for the + protected document, or recovery/Safe Mode/activation) blocking COM as the + cause and recommended surfacing it (run visible) + supplying the password + / clearing the prompt - without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/0162000c28.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/0162000c28.json deleted file mode 100644 index cffdd3f2bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/0162000c28.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415", - "State": "Stopped", - "ProcessName": "NDAFiller", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-11T10:05:02.110Z", - "EndTime": "2026-06-11T10:35:09.730Z", - "CreationTime": "2026-06-11T10:05:01.880Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/0b0669cfea.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/0b0669cfea.json deleted file mode 100644 index 2a44b28d0b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/0b0669cfea.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304", - "Name": "Legal Docs", - "Path": "Legal Docs", - "Description": "Production folder for legal document automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/3b3b2e59a8.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/3b3b2e59a8.json deleted file mode 100644 index 8b0225acaf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/3b3b2e59a8.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-11T10:05:02.160Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T10:05:03.940Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-11T10:05:04.520Z", - "Message": "[Word Application Scope] Opening document 'data\\NDA-Protected.docx'" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-11T10:35:09.700Z", - "Message": "Execution was canceled. The job ran for 00:30:07 and never advanced past [Word Application Scope]." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/3fb9a5fa3a.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/3fb9a5fa3a.json deleted file mode 100644 index 94f7d27f1b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/3fb9a5fa3a.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415", - "StartTime": "2026-06-11T10:05:02.110Z", - "EndTime": "2026-06-11T10:35:09.730Z", - "State": "Stopped", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b8c9d0e1-f2a3-4315-c6d7-e8f901243526", - "Info": "Job was canceled by user 'user1' after running for 00:30:07 with no progress. Last activity reached: Word Application Scope.", - "CreationTime": "2026-06-11T10:05:01.880Z", - "ReleaseName": "NDAFiller", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020788, - "OrganizationUnitId": 7791618, - "FolderKey": "f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-11T10:35:09.730Z", - "JobError": {}, - "ErrorCode": "", - "TraceId": "a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415", - "Id": 682425517 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/README.md deleted file mode 100644 index 1d4669118b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/README.md +++ /dev/null @@ -1,48 +0,0 @@ -# Word Application Scope Failure - Workflow Hangs on a Background Dialog - -This scenario reproduces a `Word Application Scope` **hang**: the scope -opens a password-protected document, Word raises a password prompt the -unattended robot cannot answer, and the modal dialog blocks all COM calls. -The job never advances past the scope and is cancelled after ~30 minutes -with no error. - -## What this scenario uncovers - -**Root Cause:** The `Word Application Scope` opens `NDA-Protected.docx` -with no `Password` and `Visible=False`. Word's background password prompt -(invisible unattended) wedges the COM calls, so the scope hangs. The tell -is the absence of any error: a blocking dialog produces a hang, not an -exception, and the logs stop at "Opening document". - -This maps to: -`references/activity-packages/word-activities/playbooks/word-scope-hangs-background-prompt.md` - -The correct agent behavior is to recognize the no-error hang as a hidden -modal dialog (most likely the password prompt for the protected document) -and recommend reproducing with Word visible, then supplying the password / -clearing the prompt - not to attempt host commands itself (the user is -off-host). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project whose `Word Application Scope` opens a protected doc with no password, Visible=False | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** There are no faulted jobs - the hung job shows as -> **Stopped** (cancelled), and `or jobs logs --level Error` is empty. The -> full log ends at "Opening document 'data\\NDA-Protected.docx'" and jumps -> to "Execution was canceled". The agent must broaden from a faulted-jobs -> query to find the stopped job and read the no-error hang as the signal. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `word-scope-hangs-background-prompt.md` -- Agent identified a hidden background Word dialog (password prompt for the - protected document, or recovery/Safe Mode/activation) blocking COM as the - cause and recommended surfacing it (run visible) + supplying the password - / clearing the prompt - without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/manifest.json deleted file mode 100644 index 724e96534f..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Application Scope hangs scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "0b0669cfea.json" - }, - { - "match": "or jobs list --folder-key \"f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304\" --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304 --state Faulted --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key \"f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304\" --state Stopped --output json", - "file": "0162000c28.json" - }, - { - "match": "or jobs list --folder-key f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304 --state Stopped --output json", - "file": "0162000c28.json" - }, - { - "match": "or jobs list --folder-key \"f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304\" --output json", - "file": "0162000c28.json" - }, - { - "match": "or jobs list --folder-key f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304 --output json", - "file": "0162000c28.json" - }, - { - "match": "or jobs get \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\" --output json", - "file": "3fb9a5fa3a.json" - }, - { - "match": "or jobs get a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415 --output json", - "file": "3fb9a5fa3a.json" - }, - { - "match": "or jobs logs \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\" --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415 --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "or jobs logs \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\" --output json", - "file": "3b3b2e59a8.json" - }, - { - "match": "or jobs logs a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415 --output json", - "file": "3b3b2e59a8.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/uip-fixture.json new file mode 100644 index 0000000000..12051fc715 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/data/uip-fixture.json @@ -0,0 +1,108 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304", + "--state", + "Stopped", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"NDAFiller\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T10:05:02.110Z\",\r\n \"EndTime\": \"2026-06-11T10:35:09.730Z\",\r\n \"CreationTime\": \"2026-06-11T10:05:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"NDAFiller\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-11T10:05:02.110Z\",\r\n \"EndTime\": \"2026-06-11T10:35:09.730Z\",\r\n \"CreationTime\": \"2026-06-11T10:05:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\",\r\n \"StartTime\": \"2026-06-11T10:05:02.110Z\",\r\n \"EndTime\": \"2026-06-11T10:35:09.730Z\",\r\n \"State\": \"Stopped\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b8c9d0e1-f2a3-4315-c6d7-e8f901243526\",\r\n \"Info\": \"Job was canceled by user 'user1' after running for 00:30:07 with no progress. Last activity reached: Word Application Scope.\",\r\n \"CreationTime\": \"2026-06-11T10:05:01.880Z\",\r\n \"ReleaseName\": \"NDAFiller\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020788,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-11T10:35:09.730Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"\",\r\n \"TraceId\": \"a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415\",\r\n \"Id\": 682425517\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a7b8c9d0-e1f2-4204-b5c6-d7e8f9012415", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T10:05:02.160Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T10:05:03.940Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-11T10:05:04.520Z\",\r\n \"Message\": \"[Word Application Scope] Opening document 'data\\\\NDA-Protected.docx'\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-11T10:35:09.700Z\",\r\n \"Message\": \"Execution was canceled. The job ran for 00:30:07 and never advanced past [Word Application Scope].\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f6a7b8c9-d0e1-4193-a4b5-c6d7e8f90304\",\r\n \"Name\": \"Legal Docs\",\r\n \"Path\": \"Legal Docs\",\r\n \"Description\": \"Production folder for legal document automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/task.yaml index c12db14171..66216827f9 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-hangs/task.yaml @@ -22,22 +22,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/README.md new file mode 100644 index 0000000000..43d05ab3b9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/README.md @@ -0,0 +1,48 @@ +# Word Application Scope Failure - Word Not Installed on the Robot Host + +This scenario reproduces a classic `Word Application Scope` failure caused +by **desktop Microsoft Word not being usable** on the execution machine. +The classic activity drives the Word Interop API, so the scope cannot +create the `Word.Application` COM object and faults at startup with +`REGDB_E_CLASSNOTREG` (0x80040154). + +## What this scenario uncovers + +**Root Cause:** The classic `Word Application Scope` +(`UiPath.Word.Activities.WordApplicationScope`) needs a registered desktop +Word install at a bitness compatible with the robot. The new unattended +robot host cannot create the `Word.Application` COM class, so the scope +faults before the document is opened. The "worked on dev, broke after the +move to the new robot" detail points at a host-environment cause, not a +workflow defect. + +This maps to: +`references/activity-packages/word-activities/playbooks/word-scope-com-not-installed.md` + +The user is framed as **off-host**, so the correct agent behavior is to +tie the HRESULT to an unusable Word install and recommend installing Word +(or fixing bitness / repairing the Office COM registration) - not to +attempt host commands. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project with a classic `Word Application Scope` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `word-scope-com-not-installed.md` +- Agent identified "desktop Word not usable on the robot host" as the cause + and recommended installing Word, fixing the Office/robot bitness mismatch, + or repairing the Office COM registration (any valid fix path scores full + marks), without fabricating host actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/0a798c64ab.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/0a798c64ab.json deleted file mode 100644 index 26415d4cd1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/0a798c64ab.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-08T09:12:04.120Z", - "Message": "Execution started" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-08T09:12:05.880Z", - "Message": "[Word Application Scope] Starting Microsoft Word" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-08T09:12:06.402Z", - "Message": "[Word Application Scope] Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG))." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-08T09:12:06.451Z", - "Message": "Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-08T09:12:06.470Z", - "Message": "Execution ended in: 00:00:02" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/33e05613f5.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/33e05613f5.json deleted file mode 100644 index 1c306c5020..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/33e05613f5.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-08T09:12:06.402Z", - "Message": "[Word Application Scope] Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG))." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-08T09:12:06.451Z", - "Message": "Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/8d5d16b9f7.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/8d5d16b9f7.json deleted file mode 100644 index f23ca3a509..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/8d5d16b9f7.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090", - "StartTime": "2026-06-08T09:12:04.090Z", - "EndTime": "2026-06-08T09:12:06.480Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f6a7b8c9-0d1e-4f2a-3b4c-5d6e7f809101", - "Info": "Word Application Scope: Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\r\n\r\nin Main.xaml\r\n at WordApplicationScope \"Word Application Scope\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).", - "CreationTime": "2026-06-08T09:12:03.700Z", - "ReleaseName": "DocGenerator", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020231, - "OrganizationUnitId": 7791618, - "FolderKey": "a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-08T09:12:06.480Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090", - "Id": 682409912 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/README.md deleted file mode 100644 index f72d553480..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Word Application Scope Failure - Word Not Installed on the Robot Host - -This scenario reproduces a classic `Word Application Scope` failure caused -by **desktop Microsoft Word not being usable** on the execution machine. -The classic activity drives the Word Interop API, so the scope cannot -create the `Word.Application` COM object and faults at startup with -`REGDB_E_CLASSNOTREG` (0x80040154). - -## What this scenario uncovers - -**Root Cause:** The classic `Word Application Scope` -(`UiPath.Word.Activities.WordApplicationScope`) needs a registered desktop -Word install at a bitness compatible with the robot. The new unattended -robot host cannot create the `Word.Application` COM class, so the scope -faults before the document is opened. The "worked on dev, broke after the -move to the new robot" detail points at a host-environment cause, not a -workflow defect. - -This maps to: -`references/activity-packages/word-activities/playbooks/word-scope-com-not-installed.md` - -The user is framed as **off-host**, so the correct agent behavior is to -tie the HRESULT to an unusable Word install and recommend installing Word -(or fixing bitness / repairing the Office COM registration) - not to -attempt host commands. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project with a classic `Word Application Scope` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `word-scope-com-not-installed.md` -- Agent identified "desktop Word not usable on the robot host" as the cause - and recommended installing Word, fixing the Office/robot bitness mismatch, - or repairing the Office COM registration (any valid fix path scores full - marks), without fabricating host actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/b335bd0032.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/b335bd0032.json deleted file mode 100644 index 80ad5a80bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/b335bd0032.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090", - "State": "Faulted", - "ProcessName": "DocGenerator", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-08T09:12:04.090Z", - "EndTime": "2026-06-08T09:12:06.480Z", - "CreationTime": "2026-06-08T09:12:03.700Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/e4403d3587.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/e4403d3587.json deleted file mode 100644 index 3e7c6e97b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/e4403d3587.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801", - "Name": "Contract Generation", - "Path": "Contract Generation", - "Description": "Production folder for document-generation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/manifest.json deleted file mode 100644 index fb9cde5755..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Application Scope not-installed scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "e4403d3587.json" - }, - { - "match": "or jobs list --folder-key \"a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801\" --state Faulted --output json", - "file": "b335bd0032.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801 --state Faulted --output json", - "file": "b335bd0032.json" - }, - { - "match": "or jobs get \"e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090\" --output json", - "file": "8d5d16b9f7.json" - }, - { - "match": "or jobs get e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090 --output json", - "file": "8d5d16b9f7.json" - }, - { - "match": "or jobs logs \"e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090\" --level Error --output json", - "file": "33e05613f5.json" - }, - { - "match": "or jobs logs e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090 --level Error --output json", - "file": "33e05613f5.json" - }, - { - "match": "or jobs logs \"e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090\" --output json", - "file": "0a798c64ab.json" - }, - { - "match": "or jobs logs e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090 --output json", - "file": "0a798c64ab.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/uip-fixture.json new file mode 100644 index 0000000000..2842823a1a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DocGenerator\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-08T09:12:04.090Z\",\r\n \"EndTime\": \"2026-06-08T09:12:06.480Z\",\r\n \"CreationTime\": \"2026-06-08T09:12:03.700Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-08T09:12:06.402Z\",\r\n \"Message\": \"[Word Application Scope] Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-08T09:12:06.451Z\",\r\n \"Message\": \"Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090\",\r\n \"StartTime\": \"2026-06-08T09:12:04.090Z\",\r\n \"EndTime\": \"2026-06-08T09:12:06.480Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f6a7b8c9-0d1e-4f2a-3b4c-5d6e7f809101\",\r\n \"Info\": \"Word Application Scope: Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\n\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\",\r\n \"CreationTime\": \"2026-06-08T09:12:03.700Z\",\r\n \"ReleaseName\": \"DocGenerator\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020231,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-08T09:12:06.480Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090\",\r\n \"Id\": 682409912\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "e5f6a7b8-9c0d-4e1f-2a3b-4c5d6e7f8090", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-08T09:12:04.120Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-08T09:12:05.880Z\",\r\n \"Message\": \"[Word Application Scope] Starting Microsoft Word\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-08T09:12:06.402Z\",\r\n \"Message\": \"[Word Application Scope] Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-08T09:12:06.451Z\",\r\n \"Message\": \"Retrieving the COM class factory for component with CLSID {000209FF-0000-0000-C000-000000000046} failed due to the following error: 80040154 Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)).\\r\\nin Main.xaml\\r\\n at WordApplicationScope \\\"Word Application Scope\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-08T09:12:06.470Z\",\r\n \"Message\": \"Execution ended in: 00:00:02\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4071-8293-a4b5c6d7e801\",\r\n \"Name\": \"Contract Generation\",\r\n \"Path\": \"Contract Generation\",\r\n \"Description\": \"Production folder for document-generation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/task.yaml index bc9ea3f85b..f855da7de8 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-not-installed/task.yaml @@ -20,22 +20,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/README.md new file mode 100644 index 0000000000..d99f6509ba --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/README.md @@ -0,0 +1,48 @@ +# Word Application Scope Failure - "Cannot create unknown type" (Missing Package) + +This scenario reproduces a `Word Application Scope` **load-time** failure on +an unattended robot whose `UiPath.Word.Activities` package version does not +match the published dependency. The process faults in under a second with +`Cannot create unknown type ... WordApplicationScope`, before any activity +executes. + +## What this scenario uncovers + +**Root Cause:** `Main.xaml` references `WordApplicationScope` from +`UiPath.Word.Activities` (pinned `[1.17.2]` in `project.json`). The robot +did not restore that package version, so the type cannot be created and the +workflow fails to load. "Works in Studio, fails only on the remote robot" +is the signature of a package restore / version-availability gap - not a +workflow-logic defect. + +This maps to: +`references/activity-packages/word-activities/playbooks/word-scope-cannot-create-unknown-type.md` + +The correct agent behavior is to tie the `XamlObjectWriterException` to a +missing/mismatched `UiPath.Word.Activities` package on the robot and hand +the user the feed/publish steps - NOT to edit the workflow or the activity. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | hand-authored UiPath project referencing `Word Application Scope`, pinning `UiPath.Word.Activities [1.17.2]` | + +> **Note on fixtures.** Fixtures here were authored from the documented +> playbook signature rather than captured from a real +> `.local/investigations/` session. The load-time fault (~0.8s, no +> `Starting Microsoft Word` log line) distinguishes it from the COM +> `REGDB_E_CLASSNOTREG` startup fault in `word-scope-not-installed`. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `word-scope-cannot-create-unknown-type.md` +- Agent identified a missing/mismatched `UiPath.Word.Activities` package on + the robot as the cause and recommended making the pinned version available + on the feed / pinning it / republishing - without editing the workflow or + fabricating actions + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/00cc208df1.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/00cc208df1.json deleted file mode 100644 index cacca629cb..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/00cc208df1.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-09T14:03:01.980Z", - "Message": "Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\r\nin Main.xaml\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/76b3557d32.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/76b3557d32.json deleted file mode 100644 index c787200ce1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/76b3557d32.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", - "Name": "Doc Automation", - "Path": "Doc Automation", - "Description": "Production folder for document-automation processes.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/README.md deleted file mode 100644 index 7b9f89d582..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Word Application Scope Failure - "Cannot create unknown type" (Missing Package) - -This scenario reproduces a `Word Application Scope` **load-time** failure on -an unattended robot whose `UiPath.Word.Activities` package version does not -match the published dependency. The process faults in under a second with -`Cannot create unknown type ... WordApplicationScope`, before any activity -executes. - -## What this scenario uncovers - -**Root Cause:** `Main.xaml` references `WordApplicationScope` from -`UiPath.Word.Activities` (pinned `[1.17.2]` in `project.json`). The robot -did not restore that package version, so the type cannot be created and the -workflow fails to load. "Works in Studio, fails only on the remote robot" -is the signature of a package restore / version-availability gap - not a -workflow-logic defect. - -This maps to: -`references/activity-packages/word-activities/playbooks/word-scope-cannot-create-unknown-type.md` - -The correct agent behavior is to tie the `XamlObjectWriterException` to a -missing/mismatched `UiPath.Word.Activities` package on the robot and hand -the user the feed/publish steps - NOT to edit the workflow or the activity. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | hand-authored UiPath project referencing `Word Application Scope`, pinning `UiPath.Word.Activities [1.17.2]` | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Fixtures here were authored from the documented -> playbook signature rather than captured from a real -> `.local/investigations/` session. The load-time fault (~0.8s, no -> `Starting Microsoft Word` log line) distinguishes it from the COM -> `REGDB_E_CLASSNOTREG` startup fault in `word-scope-not-installed`. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `word-scope-cannot-create-unknown-type.md` -- Agent identified a missing/mismatched `UiPath.Word.Activities` package on - the robot as the cause and recommended making the pinned version available - on the feed / pinning it / republishing - without editing the workflow or - fabricating actions diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/baff10595f.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/baff10595f.json deleted file mode 100644 index 23d0e28455..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/baff10595f.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213", - "State": "Faulted", - "ProcessName": "InvoiceLetters", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-ROBOT-02", - "MachineKey": "", - "StartTime": "2026-06-09T14:03:01.220Z", - "EndTime": "2026-06-09T14:03:02.060Z", - "CreationTime": "2026-06-09T14:03:00.980Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/bec228dc73.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/bec228dc73.json deleted file mode 100644 index bad1e50e42..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/bec228dc73.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-09T14:03:01.260Z", - "Message": "Execution started" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-09T14:03:01.980Z", - "Message": "Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\r\nin Main.xaml\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-09T14:03:02.040Z", - "Message": "Execution ended in: 00:00:00" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/bf20eb3adc.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/bf20eb3adc.json deleted file mode 100644 index 31405cd26b..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/bf20eb3adc.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213", - "StartTime": "2026-06-09T14:03:01.220Z", - "EndTime": "2026-06-09T14:03:02.060Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d4e5f6a7-b8c9-4304-b5c6-d7e8f9021324", - "Info": "Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\r\n\r\nin Main.xaml\r\n at Main \"Main\"\r\n\r\nSystem.Xaml.XamlObjectWriterException: Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.", - "CreationTime": "2026-06-09T14:03:00.980Z", - "ReleaseName": "InvoiceLetters", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-ROBOT-02", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 6020477, - "OrganizationUnitId": 7791618, - "FolderKey": "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_ROBOT", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-09T14:03:02.060Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213", - "Id": 682414408 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/manifest.json deleted file mode 100644 index d764464223..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the Word Application Scope unknown-type scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "76b3557d32.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\" --state Faulted --output json", - "file": "baff10595f.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902 --state Faulted --output json", - "file": "baff10595f.json" - }, - { - "match": "or jobs get \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213\" --output json", - "file": "bf20eb3adc.json" - }, - { - "match": "or jobs get c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213 --output json", - "file": "bf20eb3adc.json" - }, - { - "match": "or jobs logs \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213\" --level Error --output json", - "file": "00cc208df1.json" - }, - { - "match": "or jobs logs c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213 --level Error --output json", - "file": "00cc208df1.json" - }, - { - "match": "or jobs logs \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213\" --output json", - "file": "bec228dc73.json" - }, - { - "match": "or jobs logs c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213 --output json", - "file": "bec228dc73.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy open-ended docsai queries to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for unmatched commands." - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/uip-fixture.json new file mode 100644 index 0000000000..1b536f2e25 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceLetters\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-ROBOT-02\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-09T14:03:01.220Z\",\r\n \"EndTime\": \"2026-06-09T14:03:02.060Z\",\r\n \"CreationTime\": \"2026-06-09T14:03:00.980Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-09T14:03:01.980Z\",\r\n \"Message\": \"Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\\r\\nin Main.xaml\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213\",\r\n \"StartTime\": \"2026-06-09T14:03:01.220Z\",\r\n \"EndTime\": \"2026-06-09T14:03:02.060Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d4e5f6a7-b8c9-4304-b5c6-d7e8f9021324\",\r\n \"Info\": \"Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\\r\\n\\r\\nin Main.xaml\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Xaml.XamlObjectWriterException: Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\",\r\n \"CreationTime\": \"2026-06-09T14:03:00.980Z\",\r\n \"ReleaseName\": \"InvoiceLetters\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-ROBOT-02\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 6020477,\r\n \"OrganizationUnitId\": 7791618,\r\n \"FolderKey\": \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_ROBOT\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-09T14:03:02.060Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213\",\r\n \"Id\": 682414408\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3d4e5f6-a7b8-4293-a4b5-c6d7e8f90213", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-09T14:03:01.260Z\",\r\n \"Message\": \"Execution started\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-09T14:03:01.980Z\",\r\n \"Message\": \"Cannot create unknown type '{clr-namespace:UiPath.Word.Activities;assembly=UiPath.Word.Activities}WordApplicationScope'.\\r\\nin Main.xaml\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-09T14:03:02.040Z\",\r\n \"Message\": \"Execution ended in: 00:00:00\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-4182-93a4-b5c6d7e8f902\",\r\n \"Name\": \"Doc Automation\",\r\n \"Path\": \"Doc Automation\",\r\n \"Description\": \"Production folder for document-automation processes.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-5b6a-4978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/task.yaml index f5aeadc057..a880cbe66e 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/word-scope-unknown-type/task.yaml @@ -21,22 +21,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + - type: template_dir + path: process + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/README.md new file mode 100644 index 0000000000..bc116a733d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/README.md @@ -0,0 +1,29 @@ +# workflowevents-app-request-trigger-connection-lost + +Faithful-replay scenario: a UiPath-App-invoked job faults at the internal `AppRequestTrigger` activity (`UiPath.WorkflowEvents.Activities`) with `System.TimeoutException` — the SignalR connection between the App and the robot never established within 60 seconds. + +## What this exercises + +The agent must recognise this as an App↔robot **channel establishment** failure (the App never connected / the hub was unreachable), not an in-workflow error, not a `HandleAppRequest` null reference, and not an `InitializeHubConnection` bootstrap failure. Disambiguation evidence: the `Apps Request Trigger` span shows `ConnectionMode = SignalR` and there is no invoke-workflow child span (no request was ever received). Tests playbook `activity-packages/workflowevents-activities/playbooks/app-request-trigger-connection-lost.md`. + +## Evidence in the fixtures + +- `or folders list` → personal workspace + Shared. +- `or jobs list` (folder-scoped + generic fallback) → the faulted `RequestListener` job is the most recent row. +- `or jobs get ` → `State: Faulted`, `Info` = `System.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting`, frames `HubConnectionService.EnsureChannelConnectedAsync` → `AppRequestTrigger.GetWorkflowRequest` → bookmark `Throw`. +- `or jobs logs --level Error` → the same exception. +- `or jobs history ` → Pending → Running → Faulted (~66s, reflecting the 60s establish wait). +- `or jobs traces ` and `traces spans get` (both forms) → RobotJob span + Apps Request Trigger span (`ConnectionMode = SignalR`). +- `docsai ask` → passthrough. + +## Provenance + +Hand-built faithful-replay. A live repro was not stageable: the WorkflowEvents activities only execute inside the UiPath Apps / Studio Web runtime, there is no CLI path to drive a UiPath App, and the package is not in a tenant feed. Built from signatures mined verbatim from the `UiPath.WorkflowEvents.Activities` source — the 60s timeout message and `Current state:` interpolation in `HubConnectionService.EnsureChannelConnectedAsync`, and the `AppRequestTrigger.GetWorkflowRequest` → bookmark `Throw` fault path. Keys are synthetic; host → `MOCK-HOST`, identities → `original_user` / `original_email@test.com` / `UIPATH\REPLACEMENT_USER`. + +## Success criteria + +`skill_triggered` + `llm_judge` against `RESOLUTION.md` (canonical lean judge, threshold 0.7). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/4acafd3e36.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/4acafd3e36.json deleted file mode 100644 index 9b456ec206..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/4acafd3e36.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-25T15:11:08.650Z", - "JobId": 4022907 - }, - { - "State": "Running", - "Timestamp": "2026-06-25T15:10:02.330Z", - "JobId": 4022907 - }, - { - "State": "Pending", - "Timestamp": "2026-06-25T15:10:01.950Z", - "JobId": 4022907 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/583da13e3e.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/583da13e3e.json deleted file mode 100644 index 44080be435..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/583da13e3e.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "StartTime": "2026-06-25T15:10:02.310Z", - "EndTime": "2026-06-25T15:11:08.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "9a1e0c47-6b22-4d8f-bc31-2e7a5d0f9b48", - "Info": "SignalR connection did not establish within 60 seconds. Current state: Connecting\r\n\r\nin Main.xaml\r\n at AppRequestTrigger\r\n at Main\r\n\r\nSystem.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting\r\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\r\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\r\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-25T15:10:01.900Z", - "ReleaseName": "RequestListener", - "Type": "Attended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Attended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 61533, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 31044, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_user", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-25T15:11:08.640Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d72c5e813a944f0b8e216c9a0f4b7d13", - "Id": 4022907 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/725e7d1c2c.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/725e7d1c2c.json deleted file mode 100644 index 3838e9ca3d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/725e7d1c2c.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "e7a1b0c44d92f308", - "TraceId": "d72c5e813a944f0b8e216c9a0f4b7d13", - "ParentId": null, - "Name": "Robot Job d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "StartTime": "2026-06-25T15:10:05.2012001Z", - "EndTime": "2026-06-25T15:11:08.4628110Z", - "Attributes": "{\"Exception\":\"System.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting\\r\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92", - "JobKey": "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T15:11:08.3671115Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - }, - { - "Id": "f8b2c1d55ea30419", - "TraceId": "d72c5e813a944f0b8e216c9a0f4b7d13", - "ParentId": "e7a1b0c44d92f308", - "Name": "Apps Request Trigger", - "StartTime": "2026-06-25T15:10:05.3320140Z", - "EndTime": "2026-06-25T15:11:08.4010522Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.WorkflowEvents.Activities.AppRequestTrigger\",\"Public.ConnectionMode\":\"SignalR\",\"Private.DisplayName\":\"Apps Request Trigger\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92", - "JobKey": "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T15:11:08.3671115Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/7fea96181a.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/7fea96181a.json deleted file mode 100644 index 19decfdb12..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/7fea96181a.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "8d1c4b22-7e90-4a3f-9b5d-1f6a2c8e0d34", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/95ff4134c4.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/95ff4134c4.json deleted file mode 100644 index 373442bfe0..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/95ff4134c4.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "State": "Faulted", - "ProcessName": "RequestListener", - "ReleaseKey": "e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-25T15:10:02.310Z", - "EndTime": "2026-06-25T15:11:08.640Z", - "CreationTime": "2026-06-25T15:10:01.900Z" - }, - { - "Key": "3b8e0a17-9d52-4c6e-a014-7f2b9c5d83e0", - "State": "Successful", - "ProcessName": "RequestListener", - "ReleaseKey": "e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-25T14:48:11.500Z", - "EndTime": "2026-06-25T14:49:55.220Z", - "CreationTime": "2026-06-25T14:48:11.180Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/99df7baa44.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/99df7baa44.json deleted file mode 100644 index 9605ae277d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/99df7baa44.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-25T15:11:08.140Z", - "Message": "Apps Request Trigger: SignalR connection did not establish within 60 seconds. Current state: Connecting\r\nin Main.xaml\r\n at AppRequestTrigger\r\n at Main\r\n\r\n--- .NET stack trace ---\r\nSystem.TimeoutException\r\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\r\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\r\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\r\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/README.md deleted file mode 100644 index 5598d8015c..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# workflowevents-app-request-trigger-connection-lost - -Faithful-replay scenario: a UiPath-App-invoked job faults at the internal `AppRequestTrigger` activity (`UiPath.WorkflowEvents.Activities`) with `System.TimeoutException` — the SignalR connection between the App and the robot never established within 60 seconds. - -## What this exercises - -The agent must recognise this as an App↔robot **channel establishment** failure (the App never connected / the hub was unreachable), not an in-workflow error, not a `HandleAppRequest` null reference, and not an `InitializeHubConnection` bootstrap failure. Disambiguation evidence: the `Apps Request Trigger` span shows `ConnectionMode = SignalR` and there is no invoke-workflow child span (no request was ever received). Tests playbook `activity-packages/workflowevents-activities/playbooks/app-request-trigger-connection-lost.md`. - -## Evidence in the fixtures - -- `or folders list` → personal workspace + Shared. -- `or jobs list` (folder-scoped + generic fallback) → the faulted `RequestListener` job is the most recent row. -- `or jobs get ` → `State: Faulted`, `Info` = `System.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting`, frames `HubConnectionService.EnsureChannelConnectedAsync` → `AppRequestTrigger.GetWorkflowRequest` → bookmark `Throw`. -- `or jobs logs --level Error` → the same exception. -- `or jobs history ` → Pending → Running → Faulted (~66s, reflecting the 60s establish wait). -- `or jobs traces ` and `traces spans get` (both forms) → RobotJob span + Apps Request Trigger span (`ConnectionMode = SignalR`). -- `docsai ask` → passthrough. - -## Provenance - -Hand-built faithful-replay. A live repro was not stageable: the WorkflowEvents activities only execute inside the UiPath Apps / Studio Web runtime, there is no CLI path to drive a UiPath App, and the package is not in a tenant feed. Built from signatures mined verbatim from the `UiPath.WorkflowEvents.Activities` source — the 60s timeout message and `Current state:` interpolation in `HubConnectionService.EnsureChannelConnectedAsync`, and the `AppRequestTrigger.GetWorkflowRequest` → bookmark `Throw` fault path. Keys are synthetic; host → `MOCK-HOST`, identities → `original_user` / `original_email@test.com` / `UIPATH\REPLACEMENT_USER`. - -## Success criteria - -`skill_triggered` + `llm_judge` against `RESOLUTION.md` (canonical lean judge, threshold 0.7). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/manifest.json deleted file mode 100644 index af44da7f45..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/m/r/manifest.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-built faithful-replay fixtures for a UiPath.WorkflowEvents.Activities AppRequestTrigger TimeoutException (SignalR connection did not establish). Signatures and stack frames are source-grounded (HubConnectionService.EnsureChannelConnectedAsync 60s timeout; AppRequestTrigger.GetWorkflowRequest -> bookmark Throw). Substring dispatch, first match wins.", - "rules": [ - { - "match": "or folders list", - "file": "7fea96181a.json" - }, - { - "match": "or jobs get d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "file": "583da13e3e.json" - }, - { - "match": "or jobs logs d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13 --level Error", - "file": "99df7baa44.json", - "_doc": "Error-level log row repeats the exception already visible in jobs get." - }, - { - "match": "or jobs logs d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "file": "99df7baa44.json", - "_doc": "The Error-level set answers any logs variant." - }, - { - "match": "or jobs history d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "file": "4acafd3e36.json", - "_doc": "Pending -> Running -> Faulted; the ~66s Running-to-Faulted gap reflects the 60s SignalR establish wait." - }, - { - "match": "or jobs traces d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "file": "725e7d1c2c.json", - "_doc": "Two spans: RobotJob (TimeoutException stack) and the Apps Request Trigger span (FullTypeName UiPath.WorkflowEvents.Activities.AppRequestTrigger, ConnectionMode=SignalR). No invoke-workflow child span \u2014 the trigger never received a request because the channel never established." - }, - { - "match": "traces spans get --job-key d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "file": "725e7d1c2c.json", - "_doc": "Alternate trace-retrieval form; same span payload." - }, - { - "match": "traces spans get d72c5e813a944f0b8e216c9a0f4b7d13", - "file": "725e7d1c2c.json", - "_doc": "Positional TraceId form; same span payload." - }, - { - "match": "or jobs list --folder-key 2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "file": "95ff4134c4.json", - "_doc": "Substring dispatch routes --state Faulted variants here too; the failing job is the most recent row." - }, - { - "match": "or jobs list", - "file": "95ff4134c4.json", - "_doc": "Fallback for flag-order variations." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough \u2014 query strings vary between runs." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "min": 1, - "description": "Agent must fetch the failing job to read the System.TimeoutException from UiPath.WorkflowEvents.Activities.AppRequestTrigger (SignalR connection did not establish within 60 seconds)." - }, - { - "pattern": "traces", - "min": 1, - "description": "Cause disambiguation: the Apps Request Trigger span shows ConnectionMode=SignalR and the absence of any invoke-workflow child span, pinning the fault to a failed channel establishment rather than an in-workflow error or a hub-bootstrap failure." - }, - { - "pattern": "or jobs logs d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "min": 0, - "description": "OPTIONAL \u2014 Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", - "min": 0, - "description": "OPTIONAL \u2014 state transitions show the ~60s establish wait before the fault." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/uip-fixture.json new file mode 100644 index 0000000000..7abc779358 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T15:11:08.140Z\",\r\n \"Message\": \"Apps Request Trigger: SignalR connection did not establish within 60 seconds. Current state: Connecting\\r\\nin Main.xaml\\r\\n at AppRequestTrigger\\r\\n at Main\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.TimeoutException\\r\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"e7a1b0c44d92f308\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"StartTime\": \"2026-06-25T15:10:05.2012001Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.4628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting\\\\r\\\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"JobKey\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T15:11:08.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"f8b2c1d55ea30419\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"ParentId\": \"e7a1b0c44d92f308\",\r\n \"Name\": \"Apps Request Trigger\",\r\n \"StartTime\": \"2026-06-25T15:10:05.3320140Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.4010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.AppRequestTrigger\\\",\\\"Public.ConnectionMode\\\":\\\"SignalR\\\",\\\"Private.DisplayName\\\":\\\"Apps Request Trigger\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"JobKey\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T15:11:08.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RequestListener\",\r\n \"ReleaseKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T15:10:02.310Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.640Z\",\r\n \"CreationTime\": \"2026-06-25T15:10:01.900Z\"\r\n },\r\n {\r\n \"Key\": \"3b8e0a17-9d52-4c6e-a014-7f2b9c5d83e0\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"RequestListener\",\r\n \"ReleaseKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T14:48:11.500Z\",\r\n \"EndTime\": \"2026-06-25T14:49:55.220Z\",\r\n \"CreationTime\": \"2026-06-25T14:48:11.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"StartTime\": \"2026-06-25T15:10:02.310Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"9a1e0c47-6b22-4d8f-bc31-2e7a5d0f9b48\",\r\n \"Info\": \"SignalR connection did not establish within 60 seconds. Current state: Connecting\\r\\n\\r\\nin Main.xaml\\r\\n at AppRequestTrigger\\r\\n at Main\\r\\n\\r\\nSystem.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting\\r\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-25T15:10:01.900Z\",\r\n \"ReleaseName\": \"RequestListener\",\r\n \"Type\": \"Attended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Attended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 61533,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 31044,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_user\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-25T15:11:08.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"Id\": 4022907\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T15:11:08.140Z\",\r\n \"Message\": \"Apps Request Trigger: SignalR connection did not establish within 60 seconds. Current state: Connecting\\r\\nin Main.xaml\\r\\n at AppRequestTrigger\\r\\n at Main\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.TimeoutException\\r\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\r\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-25T15:11:08.650Z\",\r\n \"JobId\": 4022907\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-25T15:10:02.330Z\",\r\n \"JobId\": 4022907\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-25T15:10:01.950Z\",\r\n \"JobId\": 4022907\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"e7a1b0c44d92f308\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"StartTime\": \"2026-06-25T15:10:05.2012001Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.4628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting\\\\r\\\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"JobKey\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T15:11:08.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"f8b2c1d55ea30419\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"ParentId\": \"e7a1b0c44d92f308\",\r\n \"Name\": \"Apps Request Trigger\",\r\n \"StartTime\": \"2026-06-25T15:10:05.3320140Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.4010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.AppRequestTrigger\\\",\\\"Public.ConnectionMode\\\":\\\"SignalR\\\",\\\"Private.DisplayName\\\":\\\"Apps Request Trigger\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"JobKey\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T15:11:08.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "d72c5e813a944f0b8e216c9a0f4b7d13" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"e7a1b0c44d92f308\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"StartTime\": \"2026-06-25T15:10:05.2012001Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.4628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.TimeoutException: SignalR connection did not establish within 60 seconds. Current state: Connecting\\\\r\\\\n at UiPath.WorkflowEvents.Activities.Services.HubConnectionService.EnsureChannelConnectedAsync(CancellationToken cancellationToken)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.GetWorkflowRequest(IExecutorRuntime executorRuntime, Action`1 sendTrigger, Bookmark bookmark, BookmarkResumptionHelper bookmarkHelper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.AppRequestTrigger.Throw(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"JobKey\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T15:11:08.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"f8b2c1d55ea30419\",\r\n \"TraceId\": \"d72c5e813a944f0b8e216c9a0f4b7d13\",\r\n \"ParentId\": \"e7a1b0c44d92f308\",\r\n \"Name\": \"Apps Request Trigger\",\r\n \"StartTime\": \"2026-06-25T15:10:05.3320140Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.4010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.AppRequestTrigger\\\",\\\"Public.ConnectionMode\\\":\\\"SignalR\\\",\\\"Private.DisplayName\\\":\\\"Apps Request Trigger\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"JobKey\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T15:11:08.3671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8d1c4b22-7e90-4a3f-9b5d-1f6a2c8e0d34\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RequestListener\",\r\n \"ReleaseKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T15:10:02.310Z\",\r\n \"EndTime\": \"2026-06-25T15:11:08.640Z\",\r\n \"CreationTime\": \"2026-06-25T15:10:01.900Z\"\r\n },\r\n {\r\n \"Key\": \"3b8e0a17-9d52-4c6e-a014-7f2b9c5d83e0\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"RequestListener\",\r\n \"ReleaseKey\": \"e8b3a1d5-9c42-4f17-b6a0-3d5e8c1f0a92\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T14:48:11.500Z\",\r\n \"EndTime\": \"2026-06-25T14:49:55.220Z\",\r\n \"CreationTime\": \"2026-06-25T14:48:11.180Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", + "min": 1, + "description": "Agent must fetch the failing job to read the System.TimeoutException from UiPath.WorkflowEvents.Activities.AppRequestTrigger (SignalR connection did not establish within 60 seconds)." + }, + { + "pattern": "traces", + "min": 1, + "description": "Cause disambiguation: the Apps Request Trigger span shows ConnectionMode=SignalR and the absence of any invoke-workflow child span, pinning the fault to a failed channel establishment rather than an in-workflow error or a hub-bootstrap failure." + }, + { + "pattern": "or jobs logs d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history d72c5e81-3a94-4f0b-8e21-6c9a0f4b7d13", + "min": 0, + "description": "OPTIONAL — state transitions show the ~60s establish wait before the fault." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/task.yaml index ec694a1e48..da255dde3f 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-app-request-trigger-connection-lost/task.yaml @@ -26,20 +26,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/README.md new file mode 100644 index 0000000000..f8537989d0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/README.md @@ -0,0 +1,29 @@ +# workflowevents-handle-app-request-null-reference + +Faithful-replay scenario: a workflow invoked by a UiPath App faults at the internal `HandleAppRequest` activity (`UiPath.WorkflowEvents.Activities`) with `System.NullReferenceException`, because a null was dereferenced **inside the App-invoked workflow** (an input argument the App did not bind). + +## What this exercises + +The agent must recognise that `HandleAppRequest` is internal Apps machinery that *runs* the invoked workflow and *surfaces* its exception — so the NRE originates in the workflow / its input contract, not in the WorkflowEvents package and not in the App↔robot SignalR channel. The disambiguation evidence is in the spans: the `Handle Apps Request` span carries `WorkflowFile = ProcessRequest.xaml`, and the faulting Assign span carries `Value = io_Records.Rows.Count`. Tests playbook `activity-packages/workflowevents-activities/playbooks/handle-app-request-null-reference.md`. + +## Evidence in the fixtures + +- `or folders list` → personal workspace + Shared. +- `or jobs list` (folder-scoped + generic fallback) → the faulted `RequestHandler` job is the most recent row. +- `or jobs get ` → `State: Faulted`, `Info` with the NRE and the `UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute` faulting frame. +- `or jobs logs --level Error` → the same exception as a single Error row. +- `or jobs history ` → Pending → Running → Faulted. +- `or jobs traces ` and `traces spans get` (both forms) → RobotJob span (NRE stack) + Handle Apps Request span (`FullTypeName`, `WorkflowFile`, `ConnectionMode`) + faulting Assign span. +- `docsai ask` → passthrough. + +## Provenance + +Hand-built faithful-replay. A live repro was not stageable: the WorkflowEvents activities only execute inside the UiPath Apps / Studio Web runtime (not a plain published process), there is no CLI path to drive a UiPath App, and the package is not in a tenant feed. Per the validation-pipeline guidance, this scenario is built from signatures and stack frames mined verbatim from the `UiPath.WorkflowEvents.Activities` source (`HandleAppRequest.EndExecute`; the workflow-executor boundary; the standard `System.NullReferenceException` framework message). Job/folder/process/org/tenant keys are synthetic; host → `MOCK-HOST`, identities → `original_user` / `original_email@test.com` / `UIPATH\REPLACEMENT_USER`. + +## Success criteria + +`skill_triggered` + `llm_judge` against `RESOLUTION.md` (canonical lean judge, threshold 0.7). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/076234dafe.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/076234dafe.json deleted file mode 100644 index e2f561f410..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/076234dafe.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-25T14:22:13.402Z", - "Message": "Handle Apps Request: Object reference not set to an instance of an object.\r\nin ProcessRequest.xaml\r\n at Assign 'Compute row count'\r\n at Sequence 'Handle request'\r\n at HandleAppRequest\r\n\r\n--- .NET stack trace ---\r\nSystem.NullReferenceException\r\n at lambda_method(Closure , ActivityContext )\r\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\r\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/546c68092d.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/546c68092d.json deleted file mode 100644 index 464236384d..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/546c68092d.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "State": "Faulted", - "ProcessName": "RequestHandler", - "ReleaseKey": "c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-25T14:22:08.114Z", - "EndTime": "2026-06-25T14:22:13.902Z", - "CreationTime": "2026-06-25T14:22:07.701Z" - }, - { - "Key": "f0927ac6-51bd-4e33-8a90-6b2c4e1d70a8", - "State": "Successful", - "ProcessName": "RequestHandler", - "ReleaseKey": "c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-25T13:55:02.220Z", - "EndTime": "2026-06-25T13:55:09.410Z", - "CreationTime": "2026-06-25T13:55:01.880Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/7fea96181a.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/7fea96181a.json deleted file mode 100644 index 19decfdb12..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/7fea96181a.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "8d1c4b22-7e90-4a3f-9b5d-1f6a2c8e0d34", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/README.md deleted file mode 100644 index d33d5162e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# workflowevents-handle-app-request-null-reference - -Faithful-replay scenario: a workflow invoked by a UiPath App faults at the internal `HandleAppRequest` activity (`UiPath.WorkflowEvents.Activities`) with `System.NullReferenceException`, because a null was dereferenced **inside the App-invoked workflow** (an input argument the App did not bind). - -## What this exercises - -The agent must recognise that `HandleAppRequest` is internal Apps machinery that *runs* the invoked workflow and *surfaces* its exception — so the NRE originates in the workflow / its input contract, not in the WorkflowEvents package and not in the App↔robot SignalR channel. The disambiguation evidence is in the spans: the `Handle Apps Request` span carries `WorkflowFile = ProcessRequest.xaml`, and the faulting Assign span carries `Value = io_Records.Rows.Count`. Tests playbook `activity-packages/workflowevents-activities/playbooks/handle-app-request-null-reference.md`. - -## Evidence in the fixtures - -- `or folders list` → personal workspace + Shared. -- `or jobs list` (folder-scoped + generic fallback) → the faulted `RequestHandler` job is the most recent row. -- `or jobs get ` → `State: Faulted`, `Info` with the NRE and the `UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute` faulting frame. -- `or jobs logs --level Error` → the same exception as a single Error row. -- `or jobs history ` → Pending → Running → Faulted. -- `or jobs traces ` and `traces spans get` (both forms) → RobotJob span (NRE stack) + Handle Apps Request span (`FullTypeName`, `WorkflowFile`, `ConnectionMode`) + faulting Assign span. -- `docsai ask` → passthrough. - -## Provenance - -Hand-built faithful-replay. A live repro was not stageable: the WorkflowEvents activities only execute inside the UiPath Apps / Studio Web runtime (not a plain published process), there is no CLI path to drive a UiPath App, and the package is not in a tenant feed. Per the validation-pipeline guidance, this scenario is built from signatures and stack frames mined verbatim from the `UiPath.WorkflowEvents.Activities` source (`HandleAppRequest.EndExecute`; the workflow-executor boundary; the standard `System.NullReferenceException` framework message). Job/folder/process/org/tenant keys are synthetic; host → `MOCK-HOST`, identities → `original_user` / `original_email@test.com` / `UIPATH\REPLACEMENT_USER`. - -## Success criteria - -`skill_triggered` + `llm_judge` against `RESOLUTION.md` (canonical lean judge, threshold 0.7). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/c674fae47d.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/c674fae47d.json deleted file mode 100644 index 37c2788d09..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/c674fae47d.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-25T14:22:13.912Z", - "JobId": 4021583 - }, - { - "State": "Running", - "Timestamp": "2026-06-25T14:22:08.130Z", - "JobId": 4021583 - }, - { - "State": "Pending", - "Timestamp": "2026-06-25T14:22:07.751Z", - "JobId": 4021583 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/e54213fe0e.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/e54213fe0e.json deleted file mode 100644 index baf359070a..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/e54213fe0e.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "a1f0c33d92b46e07", - "TraceId": "b41d9f072e6a4c8b9d3f5a1e7c0b2d49", - "ParentId": null, - "Name": "Robot Job b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "StartTime": "2026-06-25T14:22:11.4012001Z", - "EndTime": "2026-06-25T14:22:13.6628110Z", - "Attributes": "{\"Exception\":\"System.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at lambda_method(Closure , ActivityContext )\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\r\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31", - "JobKey": "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T14:22:13.5012110Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - }, - { - "Id": "b2e1d44ea3c57f18", - "TraceId": "b41d9f072e6a4c8b9d3f5a1e7c0b2d49", - "ParentId": "a1f0c33d92b46e07", - "Name": "Handle Apps Request", - "StartTime": "2026-06-25T14:22:11.5320140Z", - "EndTime": "2026-06-25T14:22:13.6010522Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.WorkflowEvents.Activities.HandleAppRequest\",\"Private.WorkflowFile\":\"ProcessRequest.xaml\",\"Public.ConnectionMode\":\"RobotJS\",\"Private.DisplayName\":\"Handle Apps Request\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31", - "JobKey": "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T14:22:13.5012110Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - }, - { - "Id": "c3d2e55fb4d68029", - "TraceId": "b41d9f072e6a4c8b9d3f5a1e7c0b2d49", - "ParentId": "b2e1d44ea3c57f18", - "Name": "Compute row count", - "StartTime": "2026-06-25T14:22:13.4900274Z", - "EndTime": "2026-06-25T14:22:13.4960532Z", - "Attributes": "{\"Public.FullTypeName\":\"System.Activities.Statements.Assign\",\"Private.To\":\"rowCount\",\"Private.Value\":\"io_Records.Rows.Count\",\"Private.DisplayName\":\"Compute row count\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31", - "JobKey": "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T14:22:13.5012110Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/f452914343.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/f452914343.json deleted file mode 100644 index 6801d6e303..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/f452914343.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "StartTime": "2026-06-25T14:22:08.114Z", - "EndTime": "2026-06-25T14:22:13.902Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "5c0a91d7-2b44-4e0e-9a6c-8d1f0b3e7c52", - "Info": "Object reference not set to an instance of an object.\r\n\r\nin ProcessRequest.xaml\r\n at Assign 'Compute row count'\r\n at Sequence 'Handle request'\r\n at HandleAppRequest\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at lambda_method(Closure , ActivityContext )\r\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\r\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\r\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-25T14:22:07.701Z", - "ReleaseName": "RequestHandler", - "Type": "Attended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Attended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 61204, - "EntryPointPath": "ProcessRequest.xaml", - "OrganizationUnitId": 31044, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_user", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-25T14:22:13.902Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b41d9f072e6a4c8b9d3f5a1e7c0b2d49", - "Id": 4021583 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/manifest.json deleted file mode 100644 index 8633b60cd9..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/m/r/manifest.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-built faithful-replay fixtures for a UiPath.WorkflowEvents.Activities HandleAppRequest NullReferenceException. Signatures and stack frames are source-grounded (UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute). Substring dispatch, first match wins \u2014 specific rules precede generic ones.", - "rules": [ - { - "match": "or folders list", - "file": "7fea96181a.json" - }, - { - "match": "or jobs get b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "file": "f452914343.json" - }, - { - "match": "or jobs logs b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49 --level Error", - "file": "076234dafe.json", - "_doc": "Error-level log row repeats the exception already visible in jobs get." - }, - { - "match": "or jobs logs b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "file": "076234dafe.json", - "_doc": "The Error-level set answers any logs variant \u2014 the exception row is the only diagnostic content." - }, - { - "match": "or jobs history b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "file": "c674fae47d.json", - "_doc": "Pending -> Running -> Faulted transition set; confirms a clean single execution, no diagnostic signal beyond jobs get." - }, - { - "match": "or jobs traces b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "file": "e54213fe0e.json", - "_doc": "Three spans: RobotJob (NullReferenceException stack with the HandleAppRequest.EndExecute faulting frame), the Handle Apps Request activity span (FullTypeName UiPath.WorkflowEvents.Activities.HandleAppRequest, WorkflowFile=ProcessRequest.xaml, ConnectionMode=RobotJS \u2014 proves this is an App-invoked workflow), and the faulting Assign span (Value io_Records.Rows.Count, the null dereference inside the invoked workflow). The WorkflowFile + Assign expression are the cause-disambiguation evidence (in-workflow null vs WorkflowEvents-package bug vs SignalR channel)." - }, - { - "match": "traces spans get --job-key b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "file": "e54213fe0e.json", - "_doc": "Alternate trace-retrieval form (platform traces command); same span payload." - }, - { - "match": "traces spans get b41d9f072e6a4c8b9d3f5a1e7c0b2d49", - "file": "e54213fe0e.json", - "_doc": "Positional TraceId (32-char hex from the job record). Same span payload." - }, - { - "match": "or jobs list --folder-key 2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "file": "546c68092d.json", - "_doc": "Substring dispatch routes --state Faulted variants here too; the failing job is the most recent row." - }, - { - "match": "or jobs list", - "file": "546c68092d.json", - "_doc": "Fallback for flag-order variations. The folder of interest is Shared; the recorded folder-scoped job list is the correct answer for any jobs-list variant." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough \u2014 query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by rules returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "min": 1, - "description": "Agent must fetch the failing job to read the System.NullReferenceException surfaced through UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute." - }, - { - "pattern": "traces", - "min": 1, - "description": "Cause disambiguation: the Handle Apps Request span carries WorkflowFile=ProcessRequest.xaml and the faulting Assign span carries Value=io_Records.Rows.Count, together pinning the null dereference to the App-invoked workflow rather than the WorkflowEvents package or the SignalR channel." - }, - { - "pattern": "or jobs logs b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "min": 0, - "description": "OPTIONAL \u2014 Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", - "min": 0, - "description": "OPTIONAL \u2014 state transitions confirm a clean single execution." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/uip-fixture.json new file mode 100644 index 0000000000..fe294c886b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T14:22:13.402Z\",\r\n \"Message\": \"Handle Apps Request: Object reference not set to an instance of an object.\\r\\nin ProcessRequest.xaml\\r\\n at Assign 'Compute row count'\\r\\n at Sequence 'Handle request'\\r\\n at HandleAppRequest\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at lambda_method(Closure , ActivityContext )\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\r\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"a1f0c33d92b46e07\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"StartTime\": \"2026-06-25T14:22:11.4012001Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.6628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.NullReferenceException: Object reference not set to an instance of an object.\\\\r\\\\n at lambda_method(Closure , ActivityContext )\\\\r\\\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"b2e1d44ea3c57f18\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": \"a1f0c33d92b46e07\",\r\n \"Name\": \"Handle Apps Request\",\r\n \"StartTime\": \"2026-06-25T14:22:11.5320140Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.6010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.HandleAppRequest\\\",\\\"Private.WorkflowFile\\\":\\\"ProcessRequest.xaml\\\",\\\"Public.ConnectionMode\\\":\\\"RobotJS\\\",\\\"Private.DisplayName\\\":\\\"Handle Apps Request\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"c3d2e55fb4d68029\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": \"b2e1d44ea3c57f18\",\r\n \"Name\": \"Compute row count\",\r\n \"StartTime\": \"2026-06-25T14:22:13.4900274Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.4960532Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"System.Activities.Statements.Assign\\\",\\\"Private.To\\\":\\\"rowCount\\\",\\\"Private.Value\\\":\\\"io_Records.Rows.Count\\\",\\\"Private.DisplayName\\\":\\\"Compute row count\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RequestHandler\",\r\n \"ReleaseKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T14:22:08.114Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.902Z\",\r\n \"CreationTime\": \"2026-06-25T14:22:07.701Z\"\r\n },\r\n {\r\n \"Key\": \"f0927ac6-51bd-4e33-8a90-6b2c4e1d70a8\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"RequestHandler\",\r\n \"ReleaseKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T13:55:02.220Z\",\r\n \"EndTime\": \"2026-06-25T13:55:09.410Z\",\r\n \"CreationTime\": \"2026-06-25T13:55:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"StartTime\": \"2026-06-25T14:22:08.114Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.902Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"5c0a91d7-2b44-4e0e-9a6c-8d1f0b3e7c52\",\r\n \"Info\": \"Object reference not set to an instance of an object.\\r\\n\\r\\nin ProcessRequest.xaml\\r\\n at Assign 'Compute row count'\\r\\n at Sequence 'Handle request'\\r\\n at HandleAppRequest\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at lambda_method(Closure , ActivityContext )\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\\r\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\r\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-25T14:22:07.701Z\",\r\n \"ReleaseName\": \"RequestHandler\",\r\n \"Type\": \"Attended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Attended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 61204,\r\n \"EntryPointPath\": \"ProcessRequest.xaml\",\r\n \"OrganizationUnitId\": 31044,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_user\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-25T14:22:13.902Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"Id\": 4021583\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T14:22:13.402Z\",\r\n \"Message\": \"Handle Apps Request: Object reference not set to an instance of an object.\\r\\nin ProcessRequest.xaml\\r\\n at Assign 'Compute row count'\\r\\n at Sequence 'Handle request'\\r\\n at HandleAppRequest\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at lambda_method(Closure , ActivityContext )\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\r\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-25T14:22:13.912Z\",\r\n \"JobId\": 4021583\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-25T14:22:08.130Z\",\r\n \"JobId\": 4021583\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-25T14:22:07.751Z\",\r\n \"JobId\": 4021583\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"a1f0c33d92b46e07\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"StartTime\": \"2026-06-25T14:22:11.4012001Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.6628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.NullReferenceException: Object reference not set to an instance of an object.\\\\r\\\\n at lambda_method(Closure , ActivityContext )\\\\r\\\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"b2e1d44ea3c57f18\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": \"a1f0c33d92b46e07\",\r\n \"Name\": \"Handle Apps Request\",\r\n \"StartTime\": \"2026-06-25T14:22:11.5320140Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.6010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.HandleAppRequest\\\",\\\"Private.WorkflowFile\\\":\\\"ProcessRequest.xaml\\\",\\\"Public.ConnectionMode\\\":\\\"RobotJS\\\",\\\"Private.DisplayName\\\":\\\"Handle Apps Request\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"c3d2e55fb4d68029\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": \"b2e1d44ea3c57f18\",\r\n \"Name\": \"Compute row count\",\r\n \"StartTime\": \"2026-06-25T14:22:13.4900274Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.4960532Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"System.Activities.Statements.Assign\\\",\\\"Private.To\\\":\\\"rowCount\\\",\\\"Private.Value\\\":\\\"io_Records.Rows.Count\\\",\\\"Private.DisplayName\\\":\\\"Compute row count\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "b41d9f072e6a4c8b9d3f5a1e7c0b2d49" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"a1f0c33d92b46e07\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"StartTime\": \"2026-06-25T14:22:11.4012001Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.6628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.NullReferenceException: Object reference not set to an instance of an object.\\\\r\\\\n at lambda_method(Closure , ActivityContext )\\\\r\\\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n at UiPath.Robot.Activities.Api.WorkflowExecutor.EndExecute(IAsyncResult result)\\\\r\\\\n at UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"b2e1d44ea3c57f18\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": \"a1f0c33d92b46e07\",\r\n \"Name\": \"Handle Apps Request\",\r\n \"StartTime\": \"2026-06-25T14:22:11.5320140Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.6010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.HandleAppRequest\\\",\\\"Private.WorkflowFile\\\":\\\"ProcessRequest.xaml\\\",\\\"Public.ConnectionMode\\\":\\\"RobotJS\\\",\\\"Private.DisplayName\\\":\\\"Handle Apps Request\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"c3d2e55fb4d68029\",\r\n \"TraceId\": \"b41d9f072e6a4c8b9d3f5a1e7c0b2d49\",\r\n \"ParentId\": \"b2e1d44ea3c57f18\",\r\n \"Name\": \"Compute row count\",\r\n \"StartTime\": \"2026-06-25T14:22:13.4900274Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.4960532Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"System.Activities.Statements.Assign\\\",\\\"Private.To\\\":\\\"rowCount\\\",\\\"Private.Value\\\":\\\"io_Records.Rows.Count\\\",\\\"Private.DisplayName\\\":\\\"Compute row count\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"JobKey\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T14:22:13.5012110Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8d1c4b22-7e90-4a3f-9b5d-1f6a2c8e0d34\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RequestHandler\",\r\n \"ReleaseKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T14:22:08.114Z\",\r\n \"EndTime\": \"2026-06-25T14:22:13.902Z\",\r\n \"CreationTime\": \"2026-06-25T14:22:07.701Z\"\r\n },\r\n {\r\n \"Key\": \"f0927ac6-51bd-4e33-8a90-6b2c4e1d70a8\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"RequestHandler\",\r\n \"ReleaseKey\": \"c7e2a9b4-3d10-4f88-a1c6-9e0b5d2f7a31\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T13:55:02.220Z\",\r\n \"EndTime\": \"2026-06-25T13:55:09.410Z\",\r\n \"CreationTime\": \"2026-06-25T13:55:01.880Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", + "min": 1, + "description": "Agent must fetch the failing job to read the System.NullReferenceException surfaced through UiPath.WorkflowEvents.Activities.HandleAppRequest.EndExecute." + }, + { + "pattern": "traces", + "min": 1, + "description": "Cause disambiguation: the Handle Apps Request span carries WorkflowFile=ProcessRequest.xaml and the faulting Assign span carries Value=io_Records.Rows.Count, together pinning the null dereference to the App-invoked workflow rather than the WorkflowEvents package or the SignalR channel." + }, + { + "pattern": "or jobs logs b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history b41d9f07-2e6a-4c8b-9d3f-5a1e7c0b2d49", + "min": 0, + "description": "OPTIONAL — state transitions confirm a clean single execution." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/task.yaml index 78cbf98a07..235ea757c0 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-handle-app-request-null-reference/task.yaml @@ -27,20 +27,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/README.md new file mode 100644 index 0000000000..d22cb65ccd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/README.md @@ -0,0 +1,29 @@ +# workflowevents-initialize-hub-connection-aggregate-failure + +Faithful-replay scenario: a Studio Web app-workflow run faults at the internal `InitializeHubConnection` activity (`UiPath.WorkflowEvents.Activities`) with `System.AggregateException` wrapping `System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: ` — the synchronous hub-connection bootstrap rejected a blank session id / resource URL. + +## What this exercises + +The agent must unwrap the `AggregateException` to the inner `WorkflowApplicationException` and recognise this as a **session/URL bootstrap** failure (blank session id / unresolved resource URL from the Studio Web app-preview context), not a SignalR socket/transport failure (the background `StartConnectionAsync` swallows those), not an in-workflow error, not a `HandleAppRequest` null reference, and not an `AppRequestTrigger` timeout. Disambiguation evidence: the `Initialize Apps Hub Connection` span shows `SessionId = ""`, and the fault is fast (~3s, not a 60s connect wait). Tests playbook `activity-packages/workflowevents-activities/playbooks/initialize-hub-connection-aggregate-failure.md`. + +## Evidence in the fixtures + +- `or folders list` → personal workspace + Shared. +- `or jobs list` (folder-scoped + generic fallback) → the faulted `AppRuntimeHost` job is the most recent row. +- `or jobs get ` → `State: Faulted`, `Info` = `System.AggregateException ... ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: `, frame `InitializeHubConnection.ExecuteAsync`. +- `or jobs logs --level Error` → the same exception. +- `or jobs history ` → Pending → Running → Faulted (~3s). +- `or jobs traces ` and `traces spans get` (both forms) → RobotJob span + Initialize Apps Hub Connection span (`SessionId = ""`). +- `docsai ask` → passthrough. + +## Provenance + +Hand-built faithful-replay. A live repro was not stageable: the WorkflowEvents activities only execute inside the UiPath Apps / Studio Web runtime, there is no CLI path to drive a Studio Web app preview into a blank-session state, and the package is not in a tenant feed. Built from signatures mined verbatim from the `UiPath.WorkflowEvents.Activities` source — the `SignalR: Invalid SessionId: {sessionId} OR Orchestrator Url: {url}` `WorkflowApplicationException` thrown synchronously in `InitializeHubConnection.ExecuteAsync`, and the source fact that the background `StartConnectionAsync` swallows transport errors (so the job-faulting `AggregateException` is the synchronous bootstrap). Keys are synthetic; host → `MOCK-HOST`, identities → `original_user` / `original_email@test.com` / `UIPATH\REPLACEMENT_USER`. + +## Success criteria + +`skill_triggered` + `llm_judge` against `RESOLUTION.md` (canonical lean judge, threshold 0.7). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/02d08506cd.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/02d08506cd.json deleted file mode 100644 index 5f05ce0ce7..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/02d08506cd.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-25T16:02:48.200Z", - "JobId": 4023415 - }, - { - "State": "Running", - "Timestamp": "2026-06-25T16:02:44.900Z", - "JobId": 4023415 - }, - { - "State": "Pending", - "Timestamp": "2026-06-25T16:02:44.520Z", - "JobId": 4023415 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/7fea96181a.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/7fea96181a.json deleted file mode 100644 index 19decfdb12..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/7fea96181a.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "8d1c4b22-7e90-4a3f-9b5d-1f6a2c8e0d34", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/README.md deleted file mode 100644 index 0635891168..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# workflowevents-initialize-hub-connection-aggregate-failure - -Faithful-replay scenario: a Studio Web app-workflow run faults at the internal `InitializeHubConnection` activity (`UiPath.WorkflowEvents.Activities`) with `System.AggregateException` wrapping `System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: ` — the synchronous hub-connection bootstrap rejected a blank session id / resource URL. - -## What this exercises - -The agent must unwrap the `AggregateException` to the inner `WorkflowApplicationException` and recognise this as a **session/URL bootstrap** failure (blank session id / unresolved resource URL from the Studio Web app-preview context), not a SignalR socket/transport failure (the background `StartConnectionAsync` swallows those), not an in-workflow error, not a `HandleAppRequest` null reference, and not an `AppRequestTrigger` timeout. Disambiguation evidence: the `Initialize Apps Hub Connection` span shows `SessionId = ""`, and the fault is fast (~3s, not a 60s connect wait). Tests playbook `activity-packages/workflowevents-activities/playbooks/initialize-hub-connection-aggregate-failure.md`. - -## Evidence in the fixtures - -- `or folders list` → personal workspace + Shared. -- `or jobs list` (folder-scoped + generic fallback) → the faulted `AppRuntimeHost` job is the most recent row. -- `or jobs get ` → `State: Faulted`, `Info` = `System.AggregateException ... ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: `, frame `InitializeHubConnection.ExecuteAsync`. -- `or jobs logs --level Error` → the same exception. -- `or jobs history ` → Pending → Running → Faulted (~3s). -- `or jobs traces ` and `traces spans get` (both forms) → RobotJob span + Initialize Apps Hub Connection span (`SessionId = ""`). -- `docsai ask` → passthrough. - -## Provenance - -Hand-built faithful-replay. A live repro was not stageable: the WorkflowEvents activities only execute inside the UiPath Apps / Studio Web runtime, there is no CLI path to drive a Studio Web app preview into a blank-session state, and the package is not in a tenant feed. Built from signatures mined verbatim from the `UiPath.WorkflowEvents.Activities` source — the `SignalR: Invalid SessionId: {sessionId} OR Orchestrator Url: {url}` `WorkflowApplicationException` thrown synchronously in `InitializeHubConnection.ExecuteAsync`, and the source fact that the background `StartConnectionAsync` swallows transport errors (so the job-faulting `AggregateException` is the synchronous bootstrap). Keys are synthetic; host → `MOCK-HOST`, identities → `original_user` / `original_email@test.com` / `UIPATH\REPLACEMENT_USER`. - -## Success criteria - -`skill_triggered` + `llm_judge` against `RESOLUTION.md` (canonical lean judge, threshold 0.7). diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/a6ac7f5705.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/a6ac7f5705.json deleted file mode 100644 index 0cdf5a4539..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/a6ac7f5705.json +++ /dev/null @@ -1,58 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "0b9e7c2a51d4f306", - "TraceId": "c0a4f7e21b584d369a728e3c5f0d2b61", - "ParentId": null, - "Name": "Robot Job c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "StartTime": "2026-06-25T16:02:47.0012001Z", - "EndTime": "2026-06-25T16:02:48.0628110Z", - "Attributes": "{\"Exception\":\"System.AggregateException: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\r\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\r\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f", - "JobKey": "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T16:02:48.1671115Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - }, - { - "Id": "1c0f8d3b62e5a417", - "TraceId": "c0a4f7e21b584d369a728e3c5f0d2b61", - "ParentId": "0b9e7c2a51d4f306", - "Name": "Initialize Apps Hub Connection", - "StartTime": "2026-06-25T16:02:47.1320140Z", - "EndTime": "2026-06-25T16:02:48.0010522Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection\",\"Private.SessionId\":\"\",\"Private.DisplayName\":\"Initialize Apps Hub Connection\"}", - "Status": 2, - "OrganizationId": "aa11bb22-cc33-4d44-ee55-ff6677889900", - "TenantId": "11aa22bb-33cc-44dd-55ee-66ff77889900", - "ExpiryTimeUtc": null, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f", - "JobKey": "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-25T16:02:48.1671115Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null, - "Context": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ca1d794009.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ca1d794009.json deleted file mode 100644 index 78b6c6f4ed..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ca1d794009.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-25T16:02:47.690Z", - "Message": "Initialize Apps Hub Connection: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\r\nin Main.xaml\r\n at InitializeHubConnection\r\n at Main\r\n\r\n--- .NET stack trace ---\r\nSystem.AggregateException\r\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \r\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ce154fba14.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ce154fba14.json deleted file mode 100644 index f82f56f57e..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ce154fba14.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "State": "Faulted", - "ProcessName": "AppRuntimeHost", - "ReleaseKey": "f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-25T16:02:44.880Z", - "EndTime": "2026-06-25T16:02:48.190Z", - "CreationTime": "2026-06-25T16:02:44.470Z" - }, - { - "Key": "5e1a9c30-7b48-4d2f-91a6-3c0e8f2b1d64", - "State": "Successful", - "ProcessName": "AppRuntimeHost", - "ReleaseKey": "f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-06-25T15:40:18.700Z", - "EndTime": "2026-06-25T15:40:25.110Z", - "CreationTime": "2026-06-25T15:40:18.360Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ead91ff026.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ead91ff026.json deleted file mode 100644 index 0424ff4aa1..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/ead91ff026.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "StartTime": "2026-06-25T16:02:44.880Z", - "EndTime": "2026-06-25T16:02:48.190Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "7d2f8a04-5c61-4b9e-8a13-0f6c2e5b9d70", - "Info": "One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\r\n\r\nin Main.xaml\r\n at InitializeHubConnection\r\n at Main\r\n\r\nSystem.AggregateException: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\r\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \r\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\r\n --- End of inner exception stack trace ---\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-25T16:02:44.470Z", - "ReleaseName": "AppRuntimeHost", - "Type": "Attended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Attended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 61702, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 31044, - "FolderKey": "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_user", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-25T16:02:48.190Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c0a4f7e21b584d369a728e3c5f0d2b61", - "Id": 4023415 - } -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/manifest.json deleted file mode 100644 index 3d9a3abe97..0000000000 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/m/r/manifest.json +++ /dev/null @@ -1,96 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-built faithful-replay fixtures for a UiPath.WorkflowEvents.Activities InitializeHubConnection AggregateException. Signatures and stack frames are source-grounded (InitializeHubConnection.ExecuteAsync throws WorkflowApplicationException 'SignalR: Invalid SessionId: ... OR Orchestrator Url: ...' synchronously; the async-activity framework surfaces it as AggregateException). Substring dispatch, first match wins.", - "rules": [ - { - "match": "or folders list", - "file": "7fea96181a.json" - }, - { - "match": "or jobs get c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "file": "ead91ff026.json" - }, - { - "match": "or jobs logs c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61 --level Error", - "file": "ca1d794009.json", - "_doc": "Error-level log row repeats the exception already visible in jobs get." - }, - { - "match": "or jobs logs c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "file": "ca1d794009.json", - "_doc": "The Error-level set answers any logs variant." - }, - { - "match": "or jobs history c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "file": "02d08506cd.json", - "_doc": "Pending -> Running -> Faulted; the activity fails fast in its synchronous bootstrap (~3s), confirming it is not a transport/connect wait." - }, - { - "match": "or jobs traces c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "file": "a6ac7f5705.json", - "_doc": "Two spans: RobotJob (AggregateException wrapping WorkflowApplicationException) and the Initialize Apps Hub Connection span (FullTypeName UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection, SessionId='' \u2014 the blank session id that triggers the synchronous throw). The blank SessionId + the inner 'Invalid SessionId OR Orchestrator Url' message are the cause-disambiguation evidence (session/URL bootstrap vs SignalR socket failure)." - }, - { - "match": "traces spans get --job-key c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "file": "a6ac7f5705.json", - "_doc": "Alternate trace-retrieval form; same span payload." - }, - { - "match": "traces spans get c0a4f7e21b584d369a728e3c5f0d2b61", - "file": "a6ac7f5705.json", - "_doc": "Positional TraceId form; same span payload." - }, - { - "match": "or jobs list --folder-key 2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82", - "file": "ce154fba14.json", - "_doc": "Substring dispatch routes --state Faulted variants here too; the failing job is the most recent row." - }, - { - "match": "or jobs list", - "file": "ce154fba14.json", - "_doc": "Fallback for flag-order variations." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "docsai queries always passthrough \u2014 query strings vary between runs." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Triage must resolve the Shared folder key before listing jobs." - }, - { - "pattern": "or jobs list", - "min": 1, - "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." - }, - { - "pattern": "or jobs get c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "min": 1, - "description": "Agent must fetch the failing job to read the System.AggregateException wrapping System.Activities.WorkflowApplicationException 'SignalR: Invalid SessionId: OR Orchestrator Url: ' from UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection." - }, - { - "pattern": "traces", - "min": 1, - "description": "Cause disambiguation: the Initialize Apps Hub Connection span shows SessionId='' (blank), pinning the fault to the synchronous session/URL bootstrap rather than a SignalR socket/transport failure (which the activity swallows on a background thread)." - }, - { - "pattern": "or jobs logs c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "min": 0, - "description": "OPTIONAL \u2014 Error-level log repeats the exception already visible in jobs get." - }, - { - "pattern": "or jobs history c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", - "min": 0, - "description": "OPTIONAL \u2014 fast (~3s) Pending -> Running -> Faulted confirms a synchronous bootstrap failure, not a connect wait." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/uip-fixture.json new file mode 100644 index 0000000000..fb07f3e3a1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/data/uip-fixture.json @@ -0,0 +1,153 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T16:02:47.690Z\",\r\n \"Message\": \"Initialize Apps Hub Connection: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\r\\nin Main.xaml\\r\\n at InitializeHubConnection\\r\\n at Main\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException\\r\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\r\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"0b9e7c2a51d4f306\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"StartTime\": \"2026-06-25T16:02:47.0012001Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.0628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.AggregateException: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\\\r\\\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\\\r\\\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"JobKey\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T16:02:48.1671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"1c0f8d3b62e5a417\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"ParentId\": \"0b9e7c2a51d4f306\",\r\n \"Name\": \"Initialize Apps Hub Connection\",\r\n \"StartTime\": \"2026-06-25T16:02:47.1320140Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.0010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection\\\",\\\"Private.SessionId\\\":\\\"\\\",\\\"Private.DisplayName\\\":\\\"Initialize Apps Hub Connection\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"JobKey\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T16:02:48.1671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AppRuntimeHost\",\r\n \"ReleaseKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T16:02:44.880Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.190Z\",\r\n \"CreationTime\": \"2026-06-25T16:02:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"5e1a9c30-7b48-4d2f-91a6-3c0e8f2b1d64\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"AppRuntimeHost\",\r\n \"ReleaseKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T15:40:18.700Z\",\r\n \"EndTime\": \"2026-06-25T15:40:25.110Z\",\r\n \"CreationTime\": \"2026-06-25T15:40:18.360Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"StartTime\": \"2026-06-25T16:02:44.880Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.190Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"7d2f8a04-5c61-4b9e-8a13-0f6c2e5b9d70\",\r\n \"Info\": \"One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\r\\n\\r\\nin Main.xaml\\r\\n at InitializeHubConnection\\r\\n at Main\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\r\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\r\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskCodeActivityImplementation.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-25T16:02:44.470Z\",\r\n \"ReleaseName\": \"AppRuntimeHost\",\r\n \"Type\": \"Attended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Attended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 61702,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 31044,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_user\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-25T16:02:48.190Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"Id\": 4023415\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T16:02:47.690Z\",\r\n \"Message\": \"Initialize Apps Hub Connection: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\r\\nin Main.xaml\\r\\n at InitializeHubConnection\\r\\n at Main\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException\\r\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\r\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\r\\n --- End of inner exception stack trace ---\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-25T16:02:48.200Z\",\r\n \"JobId\": 4023415\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-25T16:02:44.900Z\",\r\n \"JobId\": 4023415\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-25T16:02:44.520Z\",\r\n \"JobId\": 4023415\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"0b9e7c2a51d4f306\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"StartTime\": \"2026-06-25T16:02:47.0012001Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.0628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.AggregateException: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\\\r\\\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\\\r\\\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"JobKey\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T16:02:48.1671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"1c0f8d3b62e5a417\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"ParentId\": \"0b9e7c2a51d4f306\",\r\n \"Name\": \"Initialize Apps Hub Connection\",\r\n \"StartTime\": \"2026-06-25T16:02:47.1320140Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.0010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection\\\",\\\"Private.SessionId\\\":\\\"\\\",\\\"Private.DisplayName\\\":\\\"Initialize Apps Hub Connection\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"JobKey\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T16:02:48.1671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "c0a4f7e21b584d369a728e3c5f0d2b61" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"0b9e7c2a51d4f306\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"StartTime\": \"2026-06-25T16:02:47.0012001Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.0628110Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.AggregateException: One or more errors occurred. (SignalR: Invalid SessionId: OR Orchestrator Url: )\\\\r\\\\n ---> System.Activities.WorkflowApplicationException: SignalR: Invalid SessionId: OR Orchestrator Url: \\\\r\\\\n at UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"JobKey\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T16:02:48.1671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n },\r\n {\r\n \"Id\": \"1c0f8d3b62e5a417\",\r\n \"TraceId\": \"c0a4f7e21b584d369a728e3c5f0d2b61\",\r\n \"ParentId\": \"0b9e7c2a51d4f306\",\r\n \"Name\": \"Initialize Apps Hub Connection\",\r\n \"StartTime\": \"2026-06-25T16:02:47.1320140Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.0010522Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection\\\",\\\"Private.SessionId\\\":\\\"\\\",\\\"Private.DisplayName\\\":\\\"Initialize Apps Hub Connection\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"aa11bb22-cc33-4d44-ee55-ff6677889900\",\r\n \"TenantId\": \"11aa22bb-33cc-44dd-55ee-66ff77889900\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"JobKey\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-25T16:02:48.1671115Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null,\r\n \"Context\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"8d1c4b22-7e90-4a3f-9b5d-1f6a2c8e0d34\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"2f7b6a10-9c34-4d18-bb5e-7a0c1e3d6f82\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AppRuntimeHost\",\r\n \"ReleaseKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T16:02:44.880Z\",\r\n \"EndTime\": \"2026-06-25T16:02:48.190Z\",\r\n \"CreationTime\": \"2026-06-25T16:02:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"5e1a9c30-7b48-4d2f-91a6-3c0e8f2b1d64\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"AppRuntimeHost\",\r\n \"ReleaseKey\": \"f1c2b3a4-5d6e-4071-8293-0a1b2c3d4e5f\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-06-25T15:40:18.700Z\",\r\n \"EndTime\": \"2026-06-25T15:40:25.110Z\",\r\n \"CreationTime\": \"2026-06-25T15:40:18.360Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Triage must resolve the Shared folder key before listing jobs." + }, + { + "pattern": "or jobs list", + "min": 1, + "description": "Agent must list jobs in folder Shared to identify the most recent (faulted) job." + }, + { + "pattern": "or jobs get c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", + "min": 1, + "description": "Agent must fetch the failing job to read the System.AggregateException wrapping System.Activities.WorkflowApplicationException 'SignalR: Invalid SessionId: OR Orchestrator Url: ' from UiPath.WorkflowEvents.Activities.NetCore.Activities.InitializeHubConnection." + }, + { + "pattern": "traces", + "min": 1, + "description": "Cause disambiguation: the Initialize Apps Hub Connection span shows SessionId='' (blank), pinning the fault to the synchronous session/URL bootstrap rather than a SignalR socket/transport failure (which the activity swallows on a background thread)." + }, + { + "pattern": "or jobs logs c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", + "min": 0, + "description": "OPTIONAL — Error-level log repeats the exception already visible in jobs get." + }, + { + "pattern": "or jobs history c0a4f7e2-1b58-4d36-9a72-8e3c5f0d2b61", + "min": 0, + "description": "OPTIONAL — fast (~3s) Pending -> Running -> Faulted confirms a synchronous bootstrap failure, not a connect wait." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/task.yaml b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/task.yaml index 0bd2fe0d0e..b98dfcad1d 100644 --- a/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/activity-packages/workflowevents-initialize-hub-connection-aggregate-failure/task.yaml @@ -26,20 +26,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/README.md b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/README.md new file mode 100644 index 0000000000..dc55d38d63 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/README.md @@ -0,0 +1,65 @@ +# Faulted ExcelO365 Troubleshooting — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation (Claude Code +session `2d419eb2-70da-46d0-9c20-770bd5c5dc03.jsonl`) where an Excel-O365 +RPA Workflow process was failing in Orchestrator. + +## What the original session uncovered + +The user reported a failing job in their personal Orchestrator folder +(numeric ID `1321239`). The agent — using the `uipath-troubleshoot` skill +— iterated through several `uip` invocations: + +1. `uip or jobs list --folder-id 1321239 …` → `ValidationError: unknown option '--folder-id'` +2. Corrected to `--folder-key 1321239` → `Failure: Invalid folder key: '1321239'. Expected a UUID …` +3. Tried `uip or folders get 1321239` → `Failure: Folder not found` +4. Tried filtering `folders list` by `Id`, `ID`, by name "personal", and the runner's username → no match +5. Tried `--folder-path "Personal/1321239"`, `"My Workspace"` → all fail + +The agent's eventual conclusion: **personal-workspace folders aren't +returned by the standard `folders list` API**, so a numeric folder ID +isn't enough — the agent needs the folder's GUID key from the user. + +The session ended with the agent stating it needed the user to supply +the personal-folder GUID before the investigation could proceed. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `RPA Workflow/`, `ExcelO365.uipx`, `SolutionStorage.json`, `resources/`, `userProfile/` | frozen snapshot of the failing process (in `process/`) | + +The `uip` mock returns the exact stdout (and exit code) that the real +`uip` returned during the original session — including the +`[ERROR] Failed to load tool test-manager-tool …` stderr noise the +real CLI was emitting, and the genuine Orchestrator error envelopes +(`{"Result": "ValidationError", …}`). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- `troubleshooting.md` exists (the agent reached *some* terminal state) +- It mentions `personal` and `folder` (correctly identified the locus) +- It mentions `GUID` (correctly identified what's needed) +- `.local/investigations/state.json` exists (the troubleshooting skill actually ran) +- The agent invoked the `uipath-troubleshoot` skill + +## Re-running the extraction + +If the source transcript is updated, regenerate the fixtures: + +```bash +python tmp/extract_uip_fixtures.py +``` + + +## Adding new scenarios + +plus per-fixture files, layered with the shared dispatcher and a process +snapshot. To add a new scenario, copy the folder structure and edit the +manifest, fixtures, and prompt. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/1a2cb9d8be.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/1a2cb9d8be.json deleted file mode 100644 index c463a1c9ee..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/1a2cb9d8be.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "86ec2bec-fcde-44e3-a793-9bd3ea0cd0ab", - "Name": "replacement_email@test.com", - "ConnectorKey": "uipath-microsoft-onedrive", - "ConnectorName": "Microsoft OneDrive & SharePoint", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Created": "2026-03-31T15:00:00.900Z", - "Updated": "2026-03-31T18:33:16.356Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/2c9858ab16.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/2c9858ab16.json deleted file mode 100644 index 1dafc9a100..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/2c9858ab16.json +++ /dev/null @@ -1,56 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "8f73e7b3705293e3", - "TraceId": "3033bce655854455aaecdb7a9f791126", - "ParentId": null, - "Name": "Robot Job 3033bce6-5585-4455-aaec-db7a9f791126", - "StartTime": "2026-03-31T10:15:55.9473443Z", - "EndTime": "2026-03-31T10:15:57.629768Z", - "Attributes": "{\"Exception\":\"UiPath.MicrosoftOffice365.Office365Exception: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetAccountAsync(CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections.SafeExecute(NativeActivityContext context)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\r\\n --- End of inner exception stack trace ---\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"}", - "Status": 2, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "6c683090-a30b-4bab-b5a3-d1c6da356e15", - "JobKey": "3033bce6-5585-4455-aaec-db7a9f791126", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-03-31T10:15:56.6032322Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - }, - { - "Id": "ba1d1dfb3acc8b76", - "TraceId": "3033bce655854455aaecdb7a9f791126", - "ParentId": "8f73e7b3705293e3", - "Name": "For Each Sheet in Workbook", - "StartTime": "2026-03-31T10:15:56.9789445Z", - "EndTime": "2026-03-31T10:15:56.9907599Z", - "Attributes": "{\"Public.FullTypeName\":\"UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections\",\"Private.CurrentItemVariableName\":\"CurrentItem\",\"Public.Connector\":\"uipath-microsoft-onedrive\",\"Private.Length\":\"totalNumberOfSheetsProcessed\",\"Private.DriveItemArgument\":{\"ItemSelectionMode\":\"ItemUrl\",\"FullPathHint\":\"\",\"ParentFullPathHint\":\"\",\"ManualEntryItemUrl\":\"IT_Services_Marketshare_2018Q2.XLSM\"},\"Public.ItemSelectionMode\":\"ItemUrl\",\"Private.ManualEntryItemUrl\":\"IT_Services_Marketshare_2018Q2.XLSM\",\"Public.ConnectionId\":\"e62228b7-ffc2-49dc-a779-29207b0d2b6f\",\"Public.UseConnectionService\":\"True\",\"Public.AuthScopesInvalid\":\"False\",\"Private.DisplayName\":\"For Each Sheet in Workbook\"}", - "Status": 1, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 4, - "SpanType": "Activity", - "ProcessKey": "6c683090-a30b-4bab-b5a3-d1c6da356e15", - "JobKey": "3033bce6-5585-4455-aaec-db7a9f791126", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-03-31T10:15:56.0585637Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/34f3fc150f.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/34f3fc150f.json deleted file mode 100644 index f1e4d69259..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/34f3fc150f.json +++ /dev/null @@ -1,97 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "State": "Faulted", - "ProcessName": "MacOS-excel-localproc", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-04-01T11:05:20.893Z", - "EndTime": "2026-04-01T11:05:25.247Z", - "CreationTime": "2026-04-01T11:05:19.137Z", - "Id": 47602501 - }, - { - "Key": "100c2f87-c1b1-4aba-9488-63ca964b951c", - "State": "Faulted", - "ProcessName": "MacOS-excel-localproc", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-04-01T07:20:39.520Z", - "EndTime": "2026-04-01T07:20:43.403Z", - "CreationTime": "2026-03-31T16:31:03.350Z", - "Id": 47591023 - }, - { - "Key": "68051472-bc1a-4188-93fc-b65dad14cd99", - "State": "Faulted", - "ProcessName": "MacOS-excel-localproc", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-03-31T16:29:09.680Z", - "EndTime": "2026-03-31T16:29:12.290Z", - "CreationTime": "2026-03-31T16:29:08.500Z", - "Id": 47590968 - }, - { - "Key": "dbb32a90-89a9-4af1-bcee-6ab43a4c7c74", - "State": "Faulted", - "ProcessName": "MacOS-excel-localproc", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-03-31T16:21:24.293Z", - "EndTime": "2026-03-31T16:21:28.030Z", - "CreationTime": "2026-03-31T16:21:23.680Z", - "Id": 47590771 - }, - { - "Key": "d8cac54f-83fc-4637-9eee-d367479b009d", - "State": "Faulted", - "ProcessName": "MacOS-excel-localproc", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-03-31T15:51:12.433Z", - "EndTime": "2026-03-31T15:51:16.310Z", - "CreationTime": "2026-03-31T15:51:11.627Z", - "Id": 47590094 - }, - { - "Key": "91e5b90e-57da-4f85-991f-48461fc71e0b", - "State": "Faulted", - "ProcessName": "RPA.Workflow", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-03-31T15:04:06.730Z", - "EndTime": "2026-03-31T15:04:16.130Z", - "CreationTime": "2026-03-31T15:04:06.060Z", - "Id": 47589186 - }, - { - "Key": "3033bce6-5585-4455-aaec-db7a9f791126", - "State": "Faulted", - "ProcessName": "RPA.Workflow", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "DESKTOP-HOME-WO", - "StartTime": "2026-03-31T10:15:49.240Z", - "EndTime": "2026-03-31T10:15:57.013Z", - "CreationTime": "2026-03-31T10:15:46.003Z", - "Id": 47583596 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/4dd6cfc7a1.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/4dd6cfc7a1.json deleted file mode 100644 index 38ef5615e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/4dd6cfc7a1.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "StartTime": "2026-04-01T11:05:20.893Z", - "EndTime": "2026-04-01T11:05:25.247Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "11cea667-b9ad-43db-9794-8f0c15200542", - "Info": "The workbook does not exist\r\n\r\nin Main.xaml\r\n at ReadRow \"ReadRow\"\r\n at Sequence \"Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: The workbook does not exist\r\n at UiPath.Excel.Activities.WorkbookActivityBase`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-04-01T11:05:19.137Z", - "ReleaseName": "MacOS-excel-localproc", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "DESKTOP-HOME-WO", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Development", - "RequiresUserInteraction": false, - "ReleaseVersionId": 968463, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1321239, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\RAZVAN.MARINOVICI", - "OrchestratorUserIdentity": "replacement_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-04-01T11:05:25.247Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "Id": 47602501 - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/4e1d1a97c8.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/4e1d1a97c8.json deleted file mode 100644 index d4a60df60d..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/4e1d1a97c8.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-04-01T11:05:25.390Z", - "JobId": 47602501 - }, - { - "State": "Running", - "Timestamp": "2026-04-01T11:05:21.020Z", - "JobId": 47602501 - }, - { - "State": "Pending", - "Timestamp": "2026-04-01T11:05:19.180Z", - "JobId": 47602501 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/63754e8d91.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/63754e8d91.json deleted file mode 100644 index ce335bf22f..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/63754e8d91.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "DocsAIAsk", - "Data": { - "query_id": "9ab85416-e2c2-4422-b877-649e6f3c0531", - "text": "UiPath Orchestrator does not provide an automatic “retry policy” concept for jobs themselves. Handling child job failures and retries must be implemented in your process design and by manually restarting faulted jobs.\n\nKey points from the docs:\n\n- A job is **Faulted** if:\n - It failed to start, or \n - The process threw an unhandled error during execution. \n Faulted jobs **must be restarted manually**. \n ([Job States – multiple versions](https://docs-staging.uipath.com/orchestrator/standalone/2025.10/user-guide/job-states))\n\n- You can **restart** any job that reached a final state (**Stopped**, **Faulted**, or **Successful**) from the Jobs list:\n - Click **More Actions** → **Restart**.\n - Adjust settings in **Start Job** window if needed.\n - Click **Start**. \n ([Managing Jobs – Restarting a Job](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs))\n\n- For analysis before retrying:\n - Use **View Details** to see why a job faulted.\n - If **Enable Recording** was on for unattended jobs, you can [download execution media](https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/managing-jobs#downloading-execution-media) to inspect the failure.\n - Use **View Logs** from **More Actions** to see job logs. \n ([Managing Jobs](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs))\n\nTo implement “retry policies” for child jobs, you’d typically:\n- Catch errors in the parent workflow.\n- Decide when to restart the child job (e.g., based on exception type or retry counter).\n- Use Orchestrator APIs or activities to start the child job again.\n", - "sources": [ - "https://docs-staging.uipath.com/orchestrator/standalone/2025.10/user-guide/job-states", - "https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/job-states", - "https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs" - ], - "disclaimers": [], - "followups": [ - "How can I design a parent workflow in Studio to automatically retry a failed child job a limited number of times?", - "How do I use Orchestrator APIs or activities to programmatically start or restart jobs from a workflow?", - "What is the best way to log and track child job failures and retries in Orchestrator for reporting?" - ] - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/README.md deleted file mode 100644 index 47d3a460b9..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# Faulted ExcelO365 Troubleshooting — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation (Claude Code -session `2d419eb2-70da-46d0-9c20-770bd5c5dc03.jsonl`) where an Excel-O365 -RPA Workflow process was failing in Orchestrator. - -## What the original session uncovered - -The user reported a failing job in their personal Orchestrator folder -(numeric ID `1321239`). The agent — using the `uipath-troubleshoot` skill -— iterated through several `uip` invocations: - -1. `uip or jobs list --folder-id 1321239 …` → `ValidationError: unknown option '--folder-id'` -2. Corrected to `--folder-key 1321239` → `Failure: Invalid folder key: '1321239'. Expected a UUID …` -3. Tried `uip or folders get 1321239` → `Failure: Folder not found` -4. Tried filtering `folders list` by `Id`, `ID`, by name "personal", and the runner's username → no match -5. Tried `--folder-path "Personal/1321239"`, `"My Workspace"` → all fail - -The agent's eventual conclusion: **personal-workspace folders aren't -returned by the standard `folders list` API**, so a numeric folder ID -isn't enough — the agent needs the folder's GUID key from the user. - -The session ended with the agent stating it needed the user to supply -the personal-folder GUID before the investigation could proceed. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../mock_template/` (manifest-driven Python dispatcher) | -| `RPA Workflow/`, `ExcelO365.uipx`, `SolutionStorage.json`, `resources/`, `userProfile/` | frozen snapshot of the failing process (in `process/`) | -| `mocks/responses/*.txt` | **real** stdout extracted verbatim from the JSONL transcript (33 fixtures) | -| `mocks/responses/manifest.json` | dispatch table mapping each command pattern to its real-recorded fixture | - -The `uip` mock returns the exact stdout (and exit code) that the real -`uip` returned during the original session — including the -`[ERROR] Failed to load tool test-manager-tool …` stderr noise the -real CLI was emitting, and the genuine Orchestrator error envelopes -(`{"Result": "ValidationError", …}`). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- `troubleshooting.md` exists (the agent reached *some* terminal state) -- It mentions `personal` and `folder` (correctly identified the locus) -- It mentions `GUID` (correctly identified what's needed) -- `.local/investigations/state.json` exists (the troubleshooting skill actually ran) -- The agent invoked the `uipath-troubleshoot` skill - -## Re-running the extraction - -If the source transcript is updated, regenerate the fixtures: - -```bash -python tmp/extract_uip_fixtures.py -``` - -This rewrites `fixtures/mocks/responses/*.txt` and `_exit_codes.json`. -The manifest is hand-curated; revisit `manifest.json` after a re-extract. - -## Adding new scenarios - -This scenario shows the per-folder pattern: `mocks/responses/manifest.json` -plus per-fixture files, layered with the shared dispatcher and a process -snapshot. To add a new scenario, copy the folder structure and edit the -manifest, fixtures, and prompt. diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/ac5c9ac7fa.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/ac5c9ac7fa.json deleted file mode 100644 index 4b76be91bf..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/ac5c9ac7fa.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "b80122b16cf05adb", - "TraceId": "dbf1d1b90a4f4a4ab8679091f42fd5e1", - "ParentId": null, - "Name": "Robot Job dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "StartTime": "2026-04-01T11:05:22.3476295Z", - "EndTime": "2026-04-01T11:05:24.566616Z", - "Attributes": "{\"Exception\":\"System.ArgumentException: The workbook does not exist\\r\\n at UiPath.Excel.Activities.WorkbookActivityBase`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "dcd6f89a-f608-46f5-801d-5681cf5db147", - "JobKey": "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-04-01T11:05:24.9303922Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/b43879949a.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/b43879949a.json deleted file mode 100644 index a7f38b8e4b..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/b43879949a.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-04-01T11:05:24.529Z", - "Message": "ReadRow: The workbook does not exist\r\nin Main.xaml\r\n at ReadRow \"ReadRow\"\r\n at Sequence \"Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/bfe87a0aa9.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/bfe87a0aa9.json deleted file mode 100644 index 423c582200..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/bfe87a0aa9.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderListCurrentUser", - "Data": [ - { - "key": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "displayName": "replacement_email@test.com's workspace", - "fullyQualifiedName": "replacement_email@test.com's workspace", - "folderType": "Personal", - "id": 1321239 - }, - { - "key": "36c6d42a-9294-4807-ac48-46781c45fb96", - "displayName": "Solution 5", - "fullyQualifiedName": "replacement_email@test.com's workspace/Solution 5", - "folderType": "Solution", - "parentId": 1321239, - "parentKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "id": 2157426 - }, - { - "key": "ea201852-4965-4a5a-bf8a-9854d3a77275", - "displayName": "Shared", - "fullyQualifiedName": "Shared", - "folderType": "Standard", - "id": 294109 - }, - { - "key": "37f64013-6d26-4942-865b-cec4df0b700f", - "displayName": "Shared2", - "fullyQualifiedName": "Shared2", - "folderType": "Standard", - "id": 951575 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/c7561f8057.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/c7561f8057.json deleted file mode 100644 index 30e429c602..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/c7561f8057.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3033bce6-5585-4455-aaec-db7a9f791126", - "StartTime": "2026-03-31T10:15:49.240Z", - "EndTime": "2026-03-31T10:15:57.013Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "fac90848-2da0-4d2c-a26c-e8e33497d557", - "Info": "Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\r\n\r\nin Main.xaml\r\n at ForEachSheetConnections \"For Each Sheet in Workbook\"\r\n at Sequence \"Sequence\"\r\n at Main \"Main\"\r\n\r\nUiPath.MicrosoftOffice365.Office365Exception: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\r\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\r\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\r\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\r\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetAccountAsync(CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections.SafeExecute(NativeActivityContext context)\r\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\r\n --- End of inner exception stack trace ---\r\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\r\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\r\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\r\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetAccountAsync(CancellationToken ct)\r\n at UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections.SafeExecute(NativeActivityContext context)\r\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\r\n --- End of inner exception stack trace ---\r\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\r\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-03-31T10:15:46.003Z", - "ReleaseName": "RPA.Workflow", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "DESKTOP-HOME-WO", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Development", - "RequiresUserInteraction": false, - "ReleaseVersionId": 967813, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1321239, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\RAZVAN.MARINOVICI", - "OrchestratorUserIdentity": "replacement_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-03-31T10:15:57.013Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3033bce6-5585-4455-aaec-db7a9f791126", - "Id": 47583596 - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/e8b9ddec9e.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/e8b9ddec9e.json deleted file mode 100644 index df88c621c3..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/e8b9ddec9e.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-03-31T10:15:57.100Z", - "JobId": 47583596 - }, - { - "State": "Running", - "Timestamp": "2026-03-31T10:15:49.353Z", - "JobId": 47583596 - }, - { - "State": "Pending", - "Timestamp": "2026-03-31T10:15:46.100Z", - "JobId": 47583596 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/eaedd498c0.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/eaedd498c0.json deleted file mode 100644 index a43c28e59e..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/eaedd498c0.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-03-31T10:15:57.609Z", - "Message": "For Each Sheet in Workbook: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\r\nin Main.xaml\r\n at ForEachSheetConnections \"For Each Sheet in Workbook\"\r\n at Sequence \"Sequence\"\r\n at Main \"Main\"\r\n" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/feeae28cca.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/feeae28cca.json deleted file mode 100644 index b44948115d..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/feeae28cca.json +++ /dev/null @@ -1,6 +0,0 @@ -[ERROR] Error pinging connection 'e62228b7-ffc2-49dc-a779-29207b0d2b6f': HTTP 404: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it -{ - "Result": "Failure", - "Message": "HTTP 404: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it", - "Instructions": "Run 'uip is connections edit ' to re-authenticate, or 'uip is connections create ' to create a new connection." -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/manifest.json deleted file mode 100644 index b558f7f8c9..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/m/r/manifest.json +++ /dev/null @@ -1,104 +0,0 @@ -{ - "version": 2, - "_doc": "Real responses captured from the original session's .local/investigations/raw/. First-match-wins: list specific patterns before generic ones. `expected_calls` declares the minimum-coverage patterns the test expects the agent to exercise (compared against .calls.jsonl by tmp/coverage_report.py after the run).", - "rules": [ - { - "match": "or jobs get 3033bce6-5585-4455-aaec-db7a9f791126", - "file": "c7561f8057.json", - "exit_code": 0 - }, - { - "match": "or jobs history 3033bce6-5585-4455-aaec-db7a9f791126", - "file": "e8b9ddec9e.json", - "exit_code": 0 - }, - { - "match": "or jobs logs 3033bce6-5585-4455-aaec-db7a9f791126", - "file": "eaedd498c0.json", - "exit_code": 0 - }, - { - "match": "or jobs traces 3033bce6-5585-4455-aaec-db7a9f791126", - "file": "2c9858ab16.json", - "exit_code": 0 - }, - { - "match": "or jobs get dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "file": "4dd6cfc7a1.json", - "exit_code": 0 - }, - { - "match": "or jobs history dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "file": "4e1d1a97c8.json", - "exit_code": 0 - }, - { - "match": "or jobs logs dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "file": "b43879949a.json", - "exit_code": 0 - }, - { - "match": "or jobs traces dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1", - "file": "ac5c9ac7fa.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0 --state Faulted", - "file": "34f3fc150f.json", - "exit_code": 0 - }, - { - "match": "or jobs list --folder-key 7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "file": "34f3fc150f.json", - "exit_code": 0 - }, - { - "match": "or folders list", - "file": "bfe87a0aa9.json", - "exit_code": 0 - }, - { - "match": "is connections ping e62228b7-ffc2-49dc-a779-29207b0d2b6f", - "file": "feeae28cca.json", - "exit_code": 1 - }, - { - "match": "is connections list --folder-key 7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "file": "1a2cb9d8be.json", - "exit_code": 0 - }, - { - "match": "is connections list", - "file": "1a2cb9d8be.json", - "exit_code": 0 - }, - { - "match": "docsai ask", - "file": "63754e8d91.json", - "exit_code": 0, - "_doc": "Catch-all: every `docsai ask` (regardless of topic) returns the same Orchestrator-focused retry guidance, mirroring the single docsai call observed in the original session. If the agent asks docsai about Integration Service it receives non-blocking but off-topic content — relying on docsai for IS guidance is not part of the success path." - } - ], - "expected_calls": [ - { - "pattern": "or folders list-current-user", - "min": 1, - "description": "Agent looks up the runner's personal folder via the documented Personal-friendly endpoint." - }, - { - "pattern": "or jobs get 3033bce6-5585-4455-aaec-db7a9f791126", - "min": 1, - "description": "Agent fetches the pinned faulted job's details to read the error message and failing activity." - }, - { - "pattern": "is connections list uipath-microsoft-onedrive", - "min": 1, - "description": "Agent lists OneDrive connections in the runner's workspace to compare against the workflow's connection ID." - }, - { - "pattern": "is connections ping e62228b7-ffc2-49dc-a779-29207b0d2b6f", - "min": 0, - "description": "OPTIONAL — Agent may ping the failing connection ID to confirm the runtime error verbatim, but a more efficient path confirms cross-workspace ownership directly from the project's connection resource file + Main.xaml. Both paths are valid; the source-file path is optimal. min:0 means tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/uip-fixture.json new file mode 100644 index 0000000000..38dd3ecf51 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/data/uip-fixture.json @@ -0,0 +1,197 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-04-01T11:05:20.893Z\",\r\n \"EndTime\": \"2026-04-01T11:05:25.247Z\",\r\n \"CreationTime\": \"2026-04-01T11:05:19.137Z\",\r\n \"Id\": 47602501\r\n },\r\n {\r\n \"Key\": \"100c2f87-c1b1-4aba-9488-63ca964b951c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-04-01T07:20:39.520Z\",\r\n \"EndTime\": \"2026-04-01T07:20:43.403Z\",\r\n \"CreationTime\": \"2026-03-31T16:31:03.350Z\",\r\n \"Id\": 47591023\r\n },\r\n {\r\n \"Key\": \"68051472-bc1a-4188-93fc-b65dad14cd99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T16:29:09.680Z\",\r\n \"EndTime\": \"2026-03-31T16:29:12.290Z\",\r\n \"CreationTime\": \"2026-03-31T16:29:08.500Z\",\r\n \"Id\": 47590968\r\n },\r\n {\r\n \"Key\": \"dbb32a90-89a9-4af1-bcee-6ab43a4c7c74\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T16:21:24.293Z\",\r\n \"EndTime\": \"2026-03-31T16:21:28.030Z\",\r\n \"CreationTime\": \"2026-03-31T16:21:23.680Z\",\r\n \"Id\": 47590771\r\n },\r\n {\r\n \"Key\": \"d8cac54f-83fc-4637-9eee-d367479b009d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T15:51:12.433Z\",\r\n \"EndTime\": \"2026-03-31T15:51:16.310Z\",\r\n \"CreationTime\": \"2026-03-31T15:51:11.627Z\",\r\n \"Id\": 47590094\r\n },\r\n {\r\n \"Key\": \"91e5b90e-57da-4f85-991f-48461fc71e0b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RPA.Workflow\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T15:04:06.730Z\",\r\n \"EndTime\": \"2026-03-31T15:04:16.130Z\",\r\n \"CreationTime\": \"2026-03-31T15:04:06.060Z\",\r\n \"Id\": 47589186\r\n },\r\n {\r\n \"Key\": \"3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RPA.Workflow\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T10:15:49.240Z\",\r\n \"EndTime\": \"2026-03-31T10:15:57.013Z\",\r\n \"CreationTime\": \"2026-03-31T10:15:46.003Z\",\r\n \"Id\": 47583596\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-04-01T11:05:20.893Z\",\r\n \"EndTime\": \"2026-04-01T11:05:25.247Z\",\r\n \"CreationTime\": \"2026-04-01T11:05:19.137Z\",\r\n \"Id\": 47602501\r\n },\r\n {\r\n \"Key\": \"100c2f87-c1b1-4aba-9488-63ca964b951c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-04-01T07:20:39.520Z\",\r\n \"EndTime\": \"2026-04-01T07:20:43.403Z\",\r\n \"CreationTime\": \"2026-03-31T16:31:03.350Z\",\r\n \"Id\": 47591023\r\n },\r\n {\r\n \"Key\": \"68051472-bc1a-4188-93fc-b65dad14cd99\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T16:29:09.680Z\",\r\n \"EndTime\": \"2026-03-31T16:29:12.290Z\",\r\n \"CreationTime\": \"2026-03-31T16:29:08.500Z\",\r\n \"Id\": 47590968\r\n },\r\n {\r\n \"Key\": \"dbb32a90-89a9-4af1-bcee-6ab43a4c7c74\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T16:21:24.293Z\",\r\n \"EndTime\": \"2026-03-31T16:21:28.030Z\",\r\n \"CreationTime\": \"2026-03-31T16:21:23.680Z\",\r\n \"Id\": 47590771\r\n },\r\n {\r\n \"Key\": \"d8cac54f-83fc-4637-9eee-d367479b009d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"MacOS-excel-localproc\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T15:51:12.433Z\",\r\n \"EndTime\": \"2026-03-31T15:51:16.310Z\",\r\n \"CreationTime\": \"2026-03-31T15:51:11.627Z\",\r\n \"Id\": 47590094\r\n },\r\n {\r\n \"Key\": \"91e5b90e-57da-4f85-991f-48461fc71e0b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RPA.Workflow\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T15:04:06.730Z\",\r\n \"EndTime\": \"2026-03-31T15:04:16.130Z\",\r\n \"CreationTime\": \"2026-03-31T15:04:06.060Z\",\r\n \"Id\": 47589186\r\n },\r\n {\r\n \"Key\": \"3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RPA.Workflow\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"DESKTOP-HOME-WO\",\r\n \"StartTime\": \"2026-03-31T10:15:49.240Z\",\r\n \"EndTime\": \"2026-03-31T10:15:57.013Z\",\r\n \"CreationTime\": \"2026-03-31T10:15:46.003Z\",\r\n \"Id\": 47583596\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"86ec2bec-fcde-44e3-a793-9bd3ea0cd0ab\",\r\n \"Name\": \"replacement_email@test.com\",\r\n \"ConnectorKey\": \"uipath-microsoft-onedrive\",\r\n \"ConnectorName\": \"Microsoft OneDrive & SharePoint\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Created\": \"2026-03-31T15:00:00.900Z\",\r\n \"Updated\": \"2026-03-31T18:33:16.356Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3033bce6-5585-4455-aaec-db7a9f791126" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"StartTime\": \"2026-03-31T10:15:49.240Z\",\r\n \"EndTime\": \"2026-03-31T10:15:57.013Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"fac90848-2da0-4d2c-a26c-e8e33497d557\",\r\n \"Info\": \"Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n\\r\\nin Main.xaml\\r\\n at ForEachSheetConnections \\\"For Each Sheet in Workbook\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.MicrosoftOffice365.Office365Exception: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetAccountAsync(CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections.SafeExecute(NativeActivityContext context)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\r\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\r\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetAccountAsync(CancellationToken ct)\\r\\n at UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections.SafeExecute(NativeActivityContext context)\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\r\\n --- End of inner exception stack trace ---\\r\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\r\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-03-31T10:15:46.003Z\",\r\n \"ReleaseName\": \"RPA.Workflow\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"DESKTOP-HOME-WO\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Development\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 967813,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1321239,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\RAZVAN.MARINOVICI\",\r\n \"OrchestratorUserIdentity\": \"replacement_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-03-31T10:15:57.013Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"Id\": 47583596\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "3033bce6-5585-4455-aaec-db7a9f791126" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-03-31T10:15:57.100Z\",\r\n \"JobId\": 47583596\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-03-31T10:15:49.353Z\",\r\n \"JobId\": 47583596\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-03-31T10:15:46.100Z\",\r\n \"JobId\": 47583596\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3033bce6-5585-4455-aaec-db7a9f791126" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-03-31T10:15:57.609Z\",\r\n \"Message\": \"For Each Sheet in Workbook: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\r\\nin Main.xaml\\r\\n at ForEachSheetConnections \\\"For Each Sheet in Workbook\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "3033bce6-5585-4455-aaec-db7a9f791126" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"8f73e7b3705293e3\",\r\n \"TraceId\": \"3033bce655854455aaecdb7a9f791126\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"StartTime\": \"2026-03-31T10:15:55.9473443Z\",\r\n \"EndTime\": \"2026-03-31T10:15:57.629768Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.MicrosoftOffice365.Office365Exception: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\\\r\\\\n ---> UiPath.MicrosoftOffice365.Office365InternalException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\\\r\\\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\r\\\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass13_0.<b__0>d.MoveNext()\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\\\r\\\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphConnectionFactory.Create(IGraphClientServiceFactory factory, IConnectionClient connectionClient, OAuthDataOptions model, CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Services.GraphServiceClientProxy.GetAccountAsync(CancellationToken ct)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections.SafeExecute(NativeActivityContext context)\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n --- End of inner exception stack trace ---\\\\r\\\\n at UiPath.MicrosoftOffice365.Activities.BaseForEachConnectionServiceActivity.Execute(NativeActivityContext context)\\\\r\\\\n at System.Activities.NativeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"6c683090-a30b-4bab-b5a3-d1c6da356e15\",\r\n \"JobKey\": \"3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-03-31T10:15:56.6032322Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n },\r\n {\r\n \"Id\": \"ba1d1dfb3acc8b76\",\r\n \"TraceId\": \"3033bce655854455aaecdb7a9f791126\",\r\n \"ParentId\": \"8f73e7b3705293e3\",\r\n \"Name\": \"For Each Sheet in Workbook\",\r\n \"StartTime\": \"2026-03-31T10:15:56.9789445Z\",\r\n \"EndTime\": \"2026-03-31T10:15:56.9907599Z\",\r\n \"Attributes\": \"{\\\"Public.FullTypeName\\\":\\\"UiPath.MicrosoftOffice365.Activities.Excel.ForEachSheetConnections\\\",\\\"Private.CurrentItemVariableName\\\":\\\"CurrentItem\\\",\\\"Public.Connector\\\":\\\"uipath-microsoft-onedrive\\\",\\\"Private.Length\\\":\\\"totalNumberOfSheetsProcessed\\\",\\\"Private.DriveItemArgument\\\":{\\\"ItemSelectionMode\\\":\\\"ItemUrl\\\",\\\"FullPathHint\\\":\\\"\\\",\\\"ParentFullPathHint\\\":\\\"\\\",\\\"ManualEntryItemUrl\\\":\\\"IT_Services_Marketshare_2018Q2.XLSM\\\"},\\\"Public.ItemSelectionMode\\\":\\\"ItemUrl\\\",\\\"Private.ManualEntryItemUrl\\\":\\\"IT_Services_Marketshare_2018Q2.XLSM\\\",\\\"Public.ConnectionId\\\":\\\"e62228b7-ffc2-49dc-a779-29207b0d2b6f\\\",\\\"Public.UseConnectionService\\\":\\\"True\\\",\\\"Public.AuthScopesInvalid\\\":\\\"False\\\",\\\"Private.DisplayName\\\":\\\"For Each Sheet in Workbook\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 4,\r\n \"SpanType\": \"Activity\",\r\n \"ProcessKey\": \"6c683090-a30b-4bab-b5a3-d1c6da356e15\",\r\n \"JobKey\": \"3033bce6-5585-4455-aaec-db7a9f791126\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-03-31T10:15:56.0585637Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1\",\r\n \"StartTime\": \"2026-04-01T11:05:20.893Z\",\r\n \"EndTime\": \"2026-04-01T11:05:25.247Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"11cea667-b9ad-43db-9794-8f0c15200542\",\r\n \"Info\": \"The workbook does not exist\\r\\n\\r\\nin Main.xaml\\r\\n at ReadRow \\\"ReadRow\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: The workbook does not exist\\r\\n at UiPath.Excel.Activities.WorkbookActivityBase`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-04-01T11:05:19.137Z\",\r\n \"ReleaseName\": \"MacOS-excel-localproc\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"DESKTOP-HOME-WO\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Development\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 968463,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1321239,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\RAZVAN.MARINOVICI\",\r\n \"OrchestratorUserIdentity\": \"replacement_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-04-01T11:05:25.247Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1\",\r\n \"Id\": 47602501\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-04-01T11:05:25.390Z\",\r\n \"JobId\": 47602501\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-04-01T11:05:21.020Z\",\r\n \"JobId\": 47602501\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-04-01T11:05:19.180Z\",\r\n \"JobId\": 47602501\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-04-01T11:05:24.529Z\",\r\n \"Message\": \"ReadRow: The workbook does not exist\\r\\nin Main.xaml\\r\\n at ReadRow \\\"ReadRow\\\"\\r\\n at Sequence \\\"Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"b80122b16cf05adb\",\r\n \"TraceId\": \"dbf1d1b90a4f4a4ab8679091f42fd5e1\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1\",\r\n \"StartTime\": \"2026-04-01T11:05:22.3476295Z\",\r\n \"EndTime\": \"2026-04-01T11:05:24.566616Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentException: The workbook does not exist\\\\r\\\\n at UiPath.Excel.Activities.WorkbookActivityBase`1.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\r\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"dcd6f89a-f608-46f5-801d-5681cf5db147\",\r\n \"JobKey\": \"dbf1d1b9-0a4f-4a4a-b867-9091f42fd5e1\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-04-01T11:05:24.9303922Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "e62228b7-ffc2-49dc-a779-29207b0d2b6f" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "[ERROR] Error pinging connection 'e62228b7-ffc2-49dc-a779-29207b0d2b6f': HTTP 404: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\r\n{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"HTTP 404: Connection [e62228b7-ffc2-49dc-a779-29207b0d2b6f] is invalid or you do not have access to it\",\r\n \"Instructions\": \"Run 'uip is connections edit ' to re-authenticate, or 'uip is connections create ' to create a new connection.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"key\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"displayName\": \"replacement_email@test.com's workspace\",\r\n \"fullyQualifiedName\": \"replacement_email@test.com's workspace\",\r\n \"folderType\": \"Personal\",\r\n \"id\": 1321239\r\n },\r\n {\r\n \"key\": \"36c6d42a-9294-4807-ac48-46781c45fb96\",\r\n \"displayName\": \"Solution 5\",\r\n \"fullyQualifiedName\": \"replacement_email@test.com's workspace/Solution 5\",\r\n \"folderType\": \"Solution\",\r\n \"parentId\": 1321239,\r\n \"parentKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"id\": 2157426\r\n },\r\n {\r\n \"key\": \"ea201852-4965-4a5a-bf8a-9854d3a77275\",\r\n \"displayName\": \"Shared\",\r\n \"fullyQualifiedName\": \"Shared\",\r\n \"folderType\": \"Standard\",\r\n \"id\": 294109\r\n },\r\n {\r\n \"key\": \"37f64013-6d26-4942-865b-cec4df0b700f\",\r\n \"displayName\": \"Shared2\",\r\n \"fullyQualifiedName\": \"Shared2\",\r\n \"folderType\": \"Standard\",\r\n \"id\": 951575\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"86ec2bec-fcde-44e3-a793-9bd3ea0cd0ab\",\r\n \"Name\": \"replacement_email@test.com\",\r\n \"ConnectorKey\": \"uipath-microsoft-onedrive\",\r\n \"ConnectorName\": \"Microsoft OneDrive & SharePoint\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Created\": \"2026-03-31T15:00:00.900Z\",\r\n \"Updated\": \"2026-03-31T18:33:16.356Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "docsai", + "ask" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"DocsAIAsk\",\r\n \"Data\": {\r\n \"query_id\": \"9ab85416-e2c2-4422-b877-649e6f3c0531\",\r\n \"text\": \"UiPath Orchestrator does not provide an automatic “retry policy” concept for jobs themselves. Handling child job failures and retries must be implemented in your process design and by manually restarting faulted jobs.\\n\\nKey points from the docs:\\n\\n- A job is **Faulted** if:\\n - It failed to start, or \\n - The process threw an unhandled error during execution. \\n Faulted jobs **must be restarted manually**. \\n ([Job States – multiple versions](https://docs-staging.uipath.com/orchestrator/standalone/2025.10/user-guide/job-states))\\n\\n- You can **restart** any job that reached a final state (**Stopped**, **Faulted**, or **Successful**) from the Jobs list:\\n - Click **More Actions** → **Restart**.\\n - Adjust settings in **Start Job** window if needed.\\n - Click **Start**. \\n ([Managing Jobs – Restarting a Job](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs))\\n\\n- For analysis before retrying:\\n - Use **View Details** to see why a job faulted.\\n - If **Enable Recording** was on for unattended jobs, you can [download execution media](https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/managing-jobs#downloading-execution-media) to inspect the failure.\\n - Use **View Logs** from **More Actions** to see job logs. \\n ([Managing Jobs](https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs))\\n\\nTo implement “retry policies” for child jobs, you’d typically:\\n- Catch errors in the parent workflow.\\n- Decide when to restart the child job (e.g., based on exception type or retry counter).\\n- Use Orchestrator APIs or activities to start the child job again.\\n\",\r\n \"sources\": [\r\n \"https://docs-staging.uipath.com/orchestrator/standalone/2025.10/user-guide/job-states\",\r\n \"https://docs-staging.uipath.com/orchestrator/automation-cloud/latest/user-guide/job-states\",\r\n \"https://docs-staging.uipath.com/orchestrator/standalone/2020.10/user-guide/managing-jobs\"\r\n ],\r\n \"disclaimers\": [],\r\n \"followups\": [\r\n \"How can I design a parent workflow in Studio to automatically retry a failed child job a limited number of times?\",\r\n \"How do I use Orchestrator APIs or activities to programmatically start or restart jobs from a workflow?\",\r\n \"What is the best way to log and track child job failures and retries in Orchestrator for reporting?\"\r\n ]\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 1, + "stderr": "{\"error\": \"unmocked command\"}\n" + }, + "expected_calls": [ + { + "pattern": "or folders list-current-user", + "min": 1, + "description": "Agent looks up the runner's personal folder via the documented Personal-friendly endpoint." + }, + { + "pattern": "or jobs get 3033bce6-5585-4455-aaec-db7a9f791126", + "min": 1, + "description": "Agent fetches the pinned faulted job's details to read the error message and failing activity." + }, + { + "pattern": "is connections list uipath-microsoft-onedrive", + "min": 1, + "description": "Agent lists OneDrive connections in the runner's workspace to compare against the workflow's connection ID." + }, + { + "pattern": "is connections ping e62228b7-ffc2-49dc-a779-29207b0d2b6f", + "min": 0, + "description": "OPTIONAL — Agent may ping the failing connection ID to confirm the runtime error verbatim, but a more efficient path confirms cross-workspace ownership directly from the project's connection resource file + Main.xaml. Both paths are valid; the source-file path is optimal. min:0 means tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/task.yaml b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/task.yaml index 78b986fae1..7af585abe8 100644 --- a/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/task.yaml +++ b/tests/tasks/uipath-troubleshoot/cross-system/faulted_excel_o365/task.yaml @@ -27,22 +27,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/README.md b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/README.md new file mode 100644 index 0000000000..60e2b79dc8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/README.md @@ -0,0 +1,36 @@ +# Rpa Preflight Asset And Connection Missing — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Root Cause:** Two missing infrastructure dependencies — asset 'PO_PricingTable' and Microsoft Outlook 365 Integration Service connection — caused a preflight check failure in the PurchaseOrderProcessing workflow. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name rpa-preflight-asset-and-connection-missing --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/0432111821.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/0432111821.json deleted file mode 100644 index c62f8066b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/0432111821.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "folders", - "Description": "Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder.", - "Usage": "uip or folders [options] [command]", - "Arguments": [], - "Options": [ - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Subcommands": [ - { - "Name": "list [options]", - "Description": "List folders the current user has access to. Use --all to list all folders in the tenant (supports filtering and sorting)." - }, - { - "Name": "create [options] ", - "Description": "Create a new folder. Folders scope processes, jobs, and resources with separate permissions. Use --parent to nest inside an existing folder." - }, - { - "Name": "get [options] ", - "Description": "Get folder details by key (GUID) or path. Returns display name, key (GUID), path, type, and permissions. Use 'folders list' to find folder keys." - }, - { - "Name": "delete [options] ", - "Description": "Delete a folder by key (GUID) or path. Fails if the folder contains entities or user associations. Move or remove contents before deleting." - }, - { - "Name": "move [options] ", - "Description": "Move a folder to a new parent in the folder hierarchy. Provide the target parent folder path or key. Use --root to move to the top level." - }, - { - "Name": "edit [options] ", - "Description": "Edit folder properties (display name, description). Use 'folders get' to see current values before editing." - }, - { - "Name": "runtimes [options] ", - "Description": "List runtime allocations for a folder. Shows how many runtime slots of each type (Unattended, Attended, NonProduction, etc.) are total, connected, and available in the specified folder. Useful for checking capacity before starting jobs." - }, - { - "Name": "help [command]", - "Description": "display help for command" - } - ], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/14d1c186b6.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/14d1c186b6.json deleted file mode 100644 index a46bc61465..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/14d1c186b6.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3ecaa4ef-a1ab-4376-ae06-d3b942362e6f", - "StartTime": "2026-05-04T12:09:55.050Z", - "EndTime": "2026-05-04T12:09:57.720Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "bce1b133-6552-489c-b659-c30403dc8098", - "Info": "See Error logs (connection and/or asset read failed)\r\n\r\nin Main.xaml\r\n at Throw \"Throw Preflight Failure\"\r\n at If \"Check Preflight\"\r\n at Sequence \"PurchaseOrderProcessing\"\r\n at Main \"Main\"\r\n\r\nSystem.Exception: See Error logs (connection and/or asset read failed)\r\n at System.Activities.Statements.Throw.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-05-04T12:09:54.753Z", - "ReleaseName": "PurchaseOrderProcessing", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 4669430, - "OrganizationUnitId": 7791617, - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-04T12:09:57.720Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3ecaa4ef-a1ab-4376-ae06-d3b942362e6f", - "Id": 670910104 - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/27179fee03.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/27179fee03.json deleted file mode 100644 index ed33a3be4a..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/27179fee03.json +++ /dev/null @@ -1,246 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3ecaa4ef-a1ab-4376-ae06-d3b942362e6f", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-04T12:09:55.050Z", - "EndTime": "2026-05-04T12:09:57.720Z", - "CreationTime": "2026-05-04T12:09:54.753Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "4e459925-7c32-41dc-bfe6-2feeb14c8893", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-28T11:35:02.827Z", - "EndTime": "2026-04-28T11:35:06.440Z", - "CreationTime": "2026-04-28T11:35:02.493Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "b5f46470-68ba-4167-a2ba-9eb73bcb0fa5", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-28T11:31:40.917Z", - "EndTime": "2026-04-28T11:31:47.163Z", - "CreationTime": "2026-04-28T11:31:40.543Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "ba72841c-5cb3-4d0f-b8a6-a52ffcb6cb9e", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:56:53.430Z", - "EndTime": "2026-04-22T09:56:56.060Z", - "CreationTime": "2026-04-22T09:56:53.213Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "bf650e0b-2158-4d38-9612-ae12a49d0699", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:43:37.360Z", - "EndTime": "2026-04-22T09:43:40.003Z", - "CreationTime": "2026-04-22T09:43:37.147Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "da415db1-6a63-43d2-8181-76774268d4ee", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:43:31.200Z", - "EndTime": "2026-04-22T09:43:33.550Z", - "CreationTime": "2026-04-22T09:43:30.950Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "1b7e3d63-f404-43da-814b-8e376d890da6", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:43:09.113Z", - "EndTime": "2026-04-22T09:43:12.340Z", - "CreationTime": "2026-04-22T09:43:08.840Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "f859d126-f7b8-4e7b-a4bf-9e1ae294c8fe", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:37:48.650Z", - "EndTime": "2026-04-22T09:37:51.850Z", - "CreationTime": "2026-04-22T09:37:48.387Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "2f60884b-602e-4d3b-85b1-b4bd7cce963d", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:33:32.533Z", - "EndTime": "2026-04-22T09:33:34.923Z", - "CreationTime": "2026-04-22T09:33:32.277Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "6e37eeb5-4eb8-4cde-9beb-a325ea34f15c", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:31:53.107Z", - "EndTime": "2026-04-22T09:31:59.733Z", - "CreationTime": "2026-04-22T09:31:52.843Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "1be55087-6176-4695-a5b5-dc291d2c3058", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:26:02.617Z", - "EndTime": "2026-04-22T09:26:04.270Z", - "CreationTime": "2026-04-22T09:26:02.323Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "f3b6273a-e824-4d9a-8494-073d36104190", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:20:52.123Z", - "EndTime": "2026-04-22T09:20:52.950Z", - "CreationTime": "2026-04-22T09:20:51.897Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "a5f506d4-c8ca-4fe4-aaf2-44085474f62d", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:03:33.220Z", - "EndTime": "2026-04-22T09:03:45.763Z", - "CreationTime": "2026-04-22T09:03:32.960Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "66b871f2-22aa-47e6-a882-76dd9310e7a6", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T08:58:23.617Z", - "EndTime": "2026-04-22T08:58:24.473Z", - "CreationTime": "2026-04-22T08:58:23.350Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - }, - { - "Key": "720851d1-7e62-4615-a32c-cbc51aa05d3d", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T08:56:29.630Z", - "EndTime": "2026-04-22T08:56:44.493Z", - "CreationTime": "2026-04-22T08:53:58.787Z", - "FolderKey": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "FolderName": "PurchaseOrderProcessing" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/2ccf867f3a.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/2ccf867f3a.json deleted file mode 100644 index aaba55fc7a..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/2ccf867f3a.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-04T12:09:57.870Z", - "Message": "Throw Preflight Failure: See Error logs (connection and/or asset read failed)\r\nin Main.xaml\r\n at Throw \"Throw Preflight Failure\"\r\n at If \"Check Preflight\"\r\n at Sequence \"PurchaseOrderProcessing\"\r\n at Main \"Main\"\r\n" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T12:09:57.853Z", - "Message": "[ReadPricingAsset] Status code: 404 (Not Found). Orchestrator response: \r\nCould not find the asset 'PO_PricingTable'. Error code: 1002" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-04T12:09:57.266Z", - "Message": "[ReadPOEmail] Connection [7d15b17a-da96-4759-85ba-f24aacbfb04b] is invalid or you do not have access to it" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/4bc4212220.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/4bc4212220.json deleted file mode 100644 index 309ccd31b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/4bc4212220.json +++ /dev/null @@ -1,100 +0,0 @@ -{ - "Result": "Success", - "Code": "Help", - "Data": { - "Command": "or", - "Description": "Manage Orchestrator jobs, processes, packages, folders, machines, licenses, users, and permissions. Supports the full automation lifecycle: deploy packages, create processes, execute jobs, provision machines, manage licenses, manage access, and monitor results.", - "Usage": "uip or [options] [command]", - "Arguments": [], - "Options": [ - { - "Flags": "-h, --help", - "Description": "display help for command" - } - ], - "Subcommands": [ - { - "Name": "attachments", - "Description": "Manage job attachments. Download files attached to jobs (screenshots, reports, data files). Attachments are tenant-scoped — no folder context needed." - }, - { - "Name": "audit-logs", - "Description": "View audit logs for the tenant. Shows who did what and when — tracks changes to users, processes, assets, jobs, and other entities." - }, - { - "Name": "calendars", - "Description": "Manage calendars (Non-Working Days). Calendars define excluded dates (holidays) used by time triggers to skip scheduled runs." - }, - { - "Name": "credential-stores", - "Description": "Manage Orchestrator credential stores (tenant-scoped secure storage backends for credentials and assets)." - }, - { - "Name": "feeds", - "Description": "Manage UiPath package feeds. Feeds are repositories that store automation packages (.nupkg). Use the feed ID with 'packages list --feed-id' to browse packages in a specific feed." - }, - { - "Name": "folders", - "Description": "Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder." - }, - { - "Name": "jobs", - "Description": "Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces." - }, - { - "Name": "licenses", - "Description": "Manage UiPath Orchestrator licenses. View license allocation and usage across robot types, list runtime and named-user license assignments, and toggle machine licensing on or off." - }, - { - "Name": "machines", - "Description": "Manage UiPath Orchestrator machines (robot host definitions). Machines define the hosts where robots execute automations, along with their license slot allocations. Machines are tenant-scoped but can be assigned to specific folders." - }, - { - "Name": "packages", - "Description": "Manage UiPath automation packages (.nupkg) in the Orchestrator feed. Packages are uploaded code artifacts. Use 'processes' to manage runnable process bindings in folders." - }, - { - "Name": "roles", - "Description": "Manage Orchestrator roles and permissions (RBAC). Roles group permissions and are assigned to users at the tenant or folder level. Tenant roles apply globally; Folder roles scope access to specific folders. Use 'roles list-roles' to find role keys, 'roles list-permissions' for grantable permissions." - }, - { - "Name": "sessions", - "Description": "View and manage Orchestrator sessions (active host connections). Sessions represent live connections between hosts and Orchestrator, including their current state, runtime type, and maintenance status. Use subcommands to list sessions, inspect machine runtimes, toggle debug mode, delete inactive sessions, or set maintenance mode." - }, - { - "Name": "processes", - "Description": "Manage UiPath processes (runnable bindings of packages to folders). Processes are what you execute with 'jobs start'. Use 'packages' to manage the uploaded automation code." - }, - { - "Name": "settings", - "Description": "Manage tenant settings. Settings are key-value pairs — use 'settings list' to discover keys. Categories and common keys: [General] Abp.Timing.TimeZone (timezone ID, e.g. 'UTC' — use 'settings timezones' to list valid IDs), Abp.Localization.DefaultLanguageName (language code, e.g. 'en', 'fr', 'de'). [Email] Alerts.Email.Enabled (true/false), Abp.Net.Mail.Smtp.Host (hostname), .Port (integer, e.g. 587), .UserName, .Password, .EnableSsl (true/false), .UseDefaultCredentials (true/false), Abp.Net.Mail.DefaultFromAddress (email address). [Robot Security] Auth.Robot.EnforceAuthenticationToken (true/false — force token auth for robots), Auth.UnattendedRobot.ForceClientCredentials (True/False), AttendedRobot.RunDisconnectedHours (integer, 0=disabled, hours a robot can run disconnected). [Deployment] DeploymentUrl (NuGet feed URL), NuGet.Packages.ApiKey (API key string), Deployment.Libraries.Url, NuGet.Activities.ApiKey, Deployment.Processes.AuthenticationType (Secure/apiKey). [Scalability] Scalability.SignalR.Enabled (true/false), Scalability.SignalR.Transport (bitmask: 1=WebSockets, 2=SSE, 4=LongPolling, 7=all), Scalability.Heartbeat.PeriodSeconds (integer, default 30, seconds between robot heartbeats), Scalability.Heartbeat.FailureThreshold (integer, default 4, missed heartbeats before marking not responding). [Triggers] Triggers.DisableWhenFailedCount (integer, default 10, consecutive failures to auto-disable), Triggers.DisableWhenFailingSinceDays (integer, default 1), Triggers.JobsCountStrategy (PERPROCESS/PERTRIGGER), QueueTrigger.ActivationFrequencyInMinutes (integer, default 30), ConnectedTriggers.AutoDisableJobFailuresThreshold (integer, default 5). [Queue] inProgressMaxNumberOfMinutes (integer, default 1440=24h, max time an item stays InProgress). [Jobs] Jobs.TerminatingJobsTimeout (integer, default 1440 minutes=24h, timeout for terminating jobs). Use 'settings execution' for structured execution settings with display names and allowed values." - }, - { - "Name": "users", - "Description": "Manage Orchestrator users (tenant-scoped). List, create, edit, and delete users. Assign users to folders with specific roles. Manage tenant-level role assignments. Use 'users list' to find user keys (GUIDs)." - }, - { - "Name": "help [command]", - "Description": "display help for command" - } - ], - "GlobalOptions": [ - { - "Flags": "--output ", - "Description": "Output format (default: json)" - }, - { - "Flags": "--output-filter ", - "Description": "JMESPath expression to filter output" - }, - { - "Flags": "--log-level ", - "Description": "Log level threshold (default: info)" - }, - { - "Flags": "--log-file ", - "Description": "Write logs to file instead of stderr" - } - ] - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/59305d0a92.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/59305d0a92.json deleted file mode 100644 index 38256809ed..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/59305d0a92.json +++ /dev/null @@ -1,102 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "22302120-d65a-4f32-bc45-0a0370124f6c", - "Name": "Debug_InvoiceApproval", - "Path": "original_email@test.com's workspace/Debug_InvoiceApproval", - "Description": "", - "Type": "DebugSolution", - "ParentKey": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c" - }, - { - "Key": "1f77e545-4388-4b96-a51f-9a0f771dc331", - "Name": "InvoiceApproval", - "Path": "original_email@test.com's workspace/InvoiceApproval", - "Description": "", - "Type": "Solution", - "ParentKey": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c" - }, - { - "Key": "445c18c8-3f11-43b6-ad44-6ff83f33af18", - "Name": "BenefitsEnrollment", - "Path": "BenefitsEnrollment", - "Description": "Demo folder for Scenario 2 - Portal That Changed Overnight (DevCon 2026 Troubleshoot Agent)", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "236eb3fa-5689-467a-9025-d57f6a0614a7", - "Name": "Demo3", - "Path": "Demo3", - "Description": "", - "Type": "Solution", - "ParentKey": "" - }, - { - "Key": "7072f52f-6e95-4ef5-b2ab-7e9ee97ea460", - "Name": "DevCon-Scenario-4", - "Path": "DevCon-Scenario-4", - "Description": "", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "4d992869-8efe-44e1-a363-733606b9b35d", - "Name": "SupportTriage", - "Path": "DevCon-Scenario-4/SupportTriage", - "Description": "", - "Type": "Solution", - "ParentKey": "7072f52f-6e95-4ef5-b2ab-7e9ee97ea460" - }, - { - "Key": "62e70ea1-11de-4263-8886-d98e3f20049c", - "Name": "InvoiceApproval", - "Path": "InvoiceApproval", - "Description": "", - "Type": "Solution", - "ParentKey": "" - }, - { - "Key": "3ac175f5-7742-4a0f-a99f-f60678c399e6", - "Name": "InvoiceApproval 1", - "Path": "InvoiceApproval 1", - "Description": "", - "Type": "Solution", - "ParentKey": "" - }, - { - "Key": "8673c3a0-e11a-418b-82d3-864200c6fb97", - "Name": "InvoiceApprovalTask", - "Path": "InvoiceApprovalTask", - "Description": "", - "Type": "Solution", - "ParentKey": "" - }, - { - "Key": "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", - "Name": "PurchaseOrderProcessing", - "Path": "PurchaseOrderProcessing", - "Description": "Demo folder for PO processing automation", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/85c0a5cc9a.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/85c0a5cc9a.json deleted file mode 100644 index db2a8b76e9..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/85c0a5cc9a.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "Message", - "Data": { - "Message": "No connections found for connection ID '7d15b17a-da96-4759-85ba-f24aacbfb04b'." - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/8b70a377b2.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/8b70a377b2.json deleted file mode 100644 index 19f09f3a7c..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/8b70a377b2.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "Message", - "Data": { - "Message": "No connections found for connector 'uipath-microsoft-outlook365'." - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/README.md deleted file mode 100644 index 62f320a6ee..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Rpa Preflight Asset And Connection Missing — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Root Cause:** Two missing infrastructure dependencies — asset 'PO_PricingTable' and Microsoft Outlook 365 Integration Service connection — caused a preflight check failure in the PurchaseOrderProcessing workflow. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name rpa-preflight-asset-and-connection-missing --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/bf63839a62.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/bf63839a62.json deleted file mode 100644 index 4e15789fd6..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/bf63839a62.json +++ /dev/null @@ -1,222 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3ecaa4ef-a1ab-4376-ae06-d3b942362e6f", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-04T12:09:55.050Z", - "EndTime": "2026-05-04T12:09:57.720Z", - "CreationTime": "2026-05-04T12:09:54.753Z" - }, - { - "Key": "4e459925-7c32-41dc-bfe6-2feeb14c8893", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-28T11:35:02.827Z", - "EndTime": "2026-04-28T11:35:06.440Z", - "CreationTime": "2026-04-28T11:35:02.493Z" - }, - { - "Key": "b5f46470-68ba-4167-a2ba-9eb73bcb0fa5", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-28T11:31:40.917Z", - "EndTime": "2026-04-28T11:31:47.163Z", - "CreationTime": "2026-04-28T11:31:40.543Z" - }, - { - "Key": "ba72841c-5cb3-4d0f-b8a6-a52ffcb6cb9e", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:56:53.430Z", - "EndTime": "2026-04-22T09:56:56.060Z", - "CreationTime": "2026-04-22T09:56:53.213Z" - }, - { - "Key": "bf650e0b-2158-4d38-9612-ae12a49d0699", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:43:37.360Z", - "EndTime": "2026-04-22T09:43:40.003Z", - "CreationTime": "2026-04-22T09:43:37.147Z" - }, - { - "Key": "da415db1-6a63-43d2-8181-76774268d4ee", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:43:31.200Z", - "EndTime": "2026-04-22T09:43:33.550Z", - "CreationTime": "2026-04-22T09:43:30.950Z" - }, - { - "Key": "1b7e3d63-f404-43da-814b-8e376d890da6", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:43:09.113Z", - "EndTime": "2026-04-22T09:43:12.340Z", - "CreationTime": "2026-04-22T09:43:08.840Z" - }, - { - "Key": "f859d126-f7b8-4e7b-a4bf-9e1ae294c8fe", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:37:48.650Z", - "EndTime": "2026-04-22T09:37:51.850Z", - "CreationTime": "2026-04-22T09:37:48.387Z" - }, - { - "Key": "2f60884b-602e-4d3b-85b1-b4bd7cce963d", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:33:32.533Z", - "EndTime": "2026-04-22T09:33:34.923Z", - "CreationTime": "2026-04-22T09:33:32.277Z" - }, - { - "Key": "6e37eeb5-4eb8-4cde-9beb-a325ea34f15c", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:31:53.107Z", - "EndTime": "2026-04-22T09:31:59.733Z", - "CreationTime": "2026-04-22T09:31:52.843Z" - }, - { - "Key": "1be55087-6176-4695-a5b5-dc291d2c3058", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:26:02.617Z", - "EndTime": "2026-04-22T09:26:04.270Z", - "CreationTime": "2026-04-22T09:26:02.323Z" - }, - { - "Key": "f3b6273a-e824-4d9a-8494-073d36104190", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:20:52.123Z", - "EndTime": "2026-04-22T09:20:52.950Z", - "CreationTime": "2026-04-22T09:20:51.897Z" - }, - { - "Key": "a5f506d4-c8ca-4fe4-aaf2-44085474f62d", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T09:03:33.220Z", - "EndTime": "2026-04-22T09:03:45.763Z", - "CreationTime": "2026-04-22T09:03:32.960Z" - }, - { - "Key": "66b871f2-22aa-47e6-a882-76dd9310e7a6", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T08:58:23.617Z", - "EndTime": "2026-04-22T08:58:24.473Z", - "CreationTime": "2026-04-22T08:58:23.350Z" - }, - { - "Key": "720851d1-7e62-4615-a32c-cbc51aa05d3d", - "State": "Faulted", - "ProcessName": "PurchaseOrderProcessing", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-04-22T08:56:29.630Z", - "EndTime": "2026-04-22T08:56:44.493Z", - "CreationTime": "2026-04-22T08:53:58.787Z" - } - ], - "Pagination": { - "Returned": 15, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/manifest.json deleted file mode 100644 index db33d6b87f..0000000000 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py — docsai rules collapsed to a single passthrough rule (per maestro-stuck-rpa-job convention); assets-list match cleaned of shell-glue suffix.", - "rules": [ - { - "match": "is connections list --folder-key 7f984a6b-1138-40fb-b48e-2c3ddf02b8f0 --output json", - "file": "8b70a377b2.json" - }, - { - "match": "is connections list --connection-id 7d15b17a-da96-4759-85ba-f24aacbfb04b --output json", - "file": "85c0a5cc9a.json" - }, - { - "match": "or folders --help", - "file": "0432111821.json" - }, - { - "match": "or folders list --output json", - "file": "59305d0a92.json" - }, - { - "match": "or jobs list --folder-key 7f984a6b-1138-40fb-b48e-2c3ddf02b8f0 --state Faulted", - "file": "bf63839a62.json" - }, - { - "match": "or jobs list --state Faulted --output json", - "file": "27179fee03.json" - }, - { - "match": "or jobs get 3ecaa4ef-a1ab-4376-ae06-d3b942362e6f", - "file": "14d1c186b6.json" - }, - { - "match": "or jobs logs 3ecaa4ef-a1ab-4376-ae06-d3b942362e6f --level Error", - "file": "2ccf867f3a.json" - }, - { - "match": "or --help", - "file": "4bc4212220.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/uip-fixture.json new file mode 100644 index 0000000000..c0aeecd451 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/data/uip-fixture.json @@ -0,0 +1,120 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3ecaa4ef-a1ab-4376-ae06-d3b942362e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-04T12:09:55.050Z\",\r\n \"EndTime\": \"2026-05-04T12:09:57.720Z\",\r\n \"CreationTime\": \"2026-05-04T12:09:54.753Z\"\r\n },\r\n {\r\n \"Key\": \"4e459925-7c32-41dc-bfe6-2feeb14c8893\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-28T11:35:02.827Z\",\r\n \"EndTime\": \"2026-04-28T11:35:06.440Z\",\r\n \"CreationTime\": \"2026-04-28T11:35:02.493Z\"\r\n },\r\n {\r\n \"Key\": \"b5f46470-68ba-4167-a2ba-9eb73bcb0fa5\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-28T11:31:40.917Z\",\r\n \"EndTime\": \"2026-04-28T11:31:47.163Z\",\r\n \"CreationTime\": \"2026-04-28T11:31:40.543Z\"\r\n },\r\n {\r\n \"Key\": \"ba72841c-5cb3-4d0f-b8a6-a52ffcb6cb9e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:56:53.430Z\",\r\n \"EndTime\": \"2026-04-22T09:56:56.060Z\",\r\n \"CreationTime\": \"2026-04-22T09:56:53.213Z\"\r\n },\r\n {\r\n \"Key\": \"bf650e0b-2158-4d38-9612-ae12a49d0699\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:43:37.360Z\",\r\n \"EndTime\": \"2026-04-22T09:43:40.003Z\",\r\n \"CreationTime\": \"2026-04-22T09:43:37.147Z\"\r\n },\r\n {\r\n \"Key\": \"da415db1-6a63-43d2-8181-76774268d4ee\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:43:31.200Z\",\r\n \"EndTime\": \"2026-04-22T09:43:33.550Z\",\r\n \"CreationTime\": \"2026-04-22T09:43:30.950Z\"\r\n },\r\n {\r\n \"Key\": \"1b7e3d63-f404-43da-814b-8e376d890da6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:43:09.113Z\",\r\n \"EndTime\": \"2026-04-22T09:43:12.340Z\",\r\n \"CreationTime\": \"2026-04-22T09:43:08.840Z\"\r\n },\r\n {\r\n \"Key\": \"f859d126-f7b8-4e7b-a4bf-9e1ae294c8fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:37:48.650Z\",\r\n \"EndTime\": \"2026-04-22T09:37:51.850Z\",\r\n \"CreationTime\": \"2026-04-22T09:37:48.387Z\"\r\n },\r\n {\r\n \"Key\": \"2f60884b-602e-4d3b-85b1-b4bd7cce963d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:33:32.533Z\",\r\n \"EndTime\": \"2026-04-22T09:33:34.923Z\",\r\n \"CreationTime\": \"2026-04-22T09:33:32.277Z\"\r\n },\r\n {\r\n \"Key\": \"6e37eeb5-4eb8-4cde-9beb-a325ea34f15c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:31:53.107Z\",\r\n \"EndTime\": \"2026-04-22T09:31:59.733Z\",\r\n \"CreationTime\": \"2026-04-22T09:31:52.843Z\"\r\n },\r\n {\r\n \"Key\": \"1be55087-6176-4695-a5b5-dc291d2c3058\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:26:02.617Z\",\r\n \"EndTime\": \"2026-04-22T09:26:04.270Z\",\r\n \"CreationTime\": \"2026-04-22T09:26:02.323Z\"\r\n },\r\n {\r\n \"Key\": \"f3b6273a-e824-4d9a-8494-073d36104190\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:20:52.123Z\",\r\n \"EndTime\": \"2026-04-22T09:20:52.950Z\",\r\n \"CreationTime\": \"2026-04-22T09:20:51.897Z\"\r\n },\r\n {\r\n \"Key\": \"a5f506d4-c8ca-4fe4-aaf2-44085474f62d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:03:33.220Z\",\r\n \"EndTime\": \"2026-04-22T09:03:45.763Z\",\r\n \"CreationTime\": \"2026-04-22T09:03:32.960Z\"\r\n },\r\n {\r\n \"Key\": \"66b871f2-22aa-47e6-a882-76dd9310e7a6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T08:58:23.617Z\",\r\n \"EndTime\": \"2026-04-22T08:58:24.473Z\",\r\n \"CreationTime\": \"2026-04-22T08:58:23.350Z\"\r\n },\r\n {\r\n \"Key\": \"720851d1-7e62-4615-a32c-cbc51aa05d3d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T08:56:29.630Z\",\r\n \"EndTime\": \"2026-04-22T08:56:44.493Z\",\r\n \"CreationTime\": \"2026-04-22T08:53:58.787Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 15,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3ecaa4ef-a1ab-4376-ae06-d3b942362e6f", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T12:09:57.870Z\",\r\n \"Message\": \"Throw Preflight Failure: See Error logs (connection and/or asset read failed)\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw Preflight Failure\\\"\\r\\n at If \\\"Check Preflight\\\"\\r\\n at Sequence \\\"PurchaseOrderProcessing\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T12:09:57.853Z\",\r\n \"Message\": \"[ReadPricingAsset] Status code: 404 (Not Found). Orchestrator response: \\r\\nCould not find the asset 'PO_PricingTable'. Error code: 1002\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-04T12:09:57.266Z\",\r\n \"Message\": \"[ReadPOEmail] Connection [7d15b17a-da96-4759-85ba-f24aacbfb04b] is invalid or you do not have access to it\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "7f984a6b-1138-40fb-b48e-2c3ddf02b8f0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Message\",\r\n \"Data\": {\r\n \"Message\": \"No connections found for connector 'uipath-microsoft-outlook365'.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--connection-id", + "7d15b17a-da96-4759-85ba-f24aacbfb04b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Message\",\r\n \"Data\": {\r\n \"Message\": \"No connections found for connection ID '7d15b17a-da96-4759-85ba-f24aacbfb04b'.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3ecaa4ef-a1ab-4376-ae06-d3b942362e6f\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-04T12:09:55.050Z\",\r\n \"EndTime\": \"2026-05-04T12:09:57.720Z\",\r\n \"CreationTime\": \"2026-05-04T12:09:54.753Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"4e459925-7c32-41dc-bfe6-2feeb14c8893\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-28T11:35:02.827Z\",\r\n \"EndTime\": \"2026-04-28T11:35:06.440Z\",\r\n \"CreationTime\": \"2026-04-28T11:35:02.493Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"b5f46470-68ba-4167-a2ba-9eb73bcb0fa5\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-28T11:31:40.917Z\",\r\n \"EndTime\": \"2026-04-28T11:31:47.163Z\",\r\n \"CreationTime\": \"2026-04-28T11:31:40.543Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"ba72841c-5cb3-4d0f-b8a6-a52ffcb6cb9e\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:56:53.430Z\",\r\n \"EndTime\": \"2026-04-22T09:56:56.060Z\",\r\n \"CreationTime\": \"2026-04-22T09:56:53.213Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"bf650e0b-2158-4d38-9612-ae12a49d0699\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:43:37.360Z\",\r\n \"EndTime\": \"2026-04-22T09:43:40.003Z\",\r\n \"CreationTime\": \"2026-04-22T09:43:37.147Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"da415db1-6a63-43d2-8181-76774268d4ee\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:43:31.200Z\",\r\n \"EndTime\": \"2026-04-22T09:43:33.550Z\",\r\n \"CreationTime\": \"2026-04-22T09:43:30.950Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"1b7e3d63-f404-43da-814b-8e376d890da6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:43:09.113Z\",\r\n \"EndTime\": \"2026-04-22T09:43:12.340Z\",\r\n \"CreationTime\": \"2026-04-22T09:43:08.840Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"f859d126-f7b8-4e7b-a4bf-9e1ae294c8fe\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:37:48.650Z\",\r\n \"EndTime\": \"2026-04-22T09:37:51.850Z\",\r\n \"CreationTime\": \"2026-04-22T09:37:48.387Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"2f60884b-602e-4d3b-85b1-b4bd7cce963d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:33:32.533Z\",\r\n \"EndTime\": \"2026-04-22T09:33:34.923Z\",\r\n \"CreationTime\": \"2026-04-22T09:33:32.277Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"6e37eeb5-4eb8-4cde-9beb-a325ea34f15c\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:31:53.107Z\",\r\n \"EndTime\": \"2026-04-22T09:31:59.733Z\",\r\n \"CreationTime\": \"2026-04-22T09:31:52.843Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"1be55087-6176-4695-a5b5-dc291d2c3058\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:26:02.617Z\",\r\n \"EndTime\": \"2026-04-22T09:26:04.270Z\",\r\n \"CreationTime\": \"2026-04-22T09:26:02.323Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"f3b6273a-e824-4d9a-8494-073d36104190\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:20:52.123Z\",\r\n \"EndTime\": \"2026-04-22T09:20:52.950Z\",\r\n \"CreationTime\": \"2026-04-22T09:20:51.897Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"a5f506d4-c8ca-4fe4-aaf2-44085474f62d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T09:03:33.220Z\",\r\n \"EndTime\": \"2026-04-22T09:03:45.763Z\",\r\n \"CreationTime\": \"2026-04-22T09:03:32.960Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"66b871f2-22aa-47e6-a882-76dd9310e7a6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T08:58:23.617Z\",\r\n \"EndTime\": \"2026-04-22T08:58:24.473Z\",\r\n \"CreationTime\": \"2026-04-22T08:58:23.350Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n },\r\n {\r\n \"Key\": \"720851d1-7e62-4615-a32c-cbc51aa05d3d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"PurchaseOrderProcessing\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-04-22T08:56:29.630Z\",\r\n \"EndTime\": \"2026-04-22T08:56:44.493Z\",\r\n \"CreationTime\": \"2026-04-22T08:53:58.787Z\",\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"FolderName\": \"PurchaseOrderProcessing\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3ecaa4ef-a1ab-4376-ae06-d3b942362e6f" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3ecaa4ef-a1ab-4376-ae06-d3b942362e6f\",\r\n \"StartTime\": \"2026-05-04T12:09:55.050Z\",\r\n \"EndTime\": \"2026-05-04T12:09:57.720Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"bce1b133-6552-489c-b659-c30403dc8098\",\r\n \"Info\": \"See Error logs (connection and/or asset read failed)\\r\\n\\r\\nin Main.xaml\\r\\n at Throw \\\"Throw Preflight Failure\\\"\\r\\n at If \\\"Check Preflight\\\"\\r\\n at Sequence \\\"PurchaseOrderProcessing\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Exception: See Error logs (connection and/or asset read failed)\\r\\n at System.Activities.Statements.Throw.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-05-04T12:09:54.753Z\",\r\n \"ReleaseName\": \"PurchaseOrderProcessing\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 4669430,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-04T12:09:57.720Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3ecaa4ef-a1ab-4376-ae06-d3b942362e6f\",\r\n \"Id\": 670910104\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"folders\",\r\n \"Description\": \"Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder.\",\r\n \"Usage\": \"uip or folders [options] [command]\",\r\n \"Arguments\": [],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Subcommands\": [\r\n {\r\n \"Name\": \"list [options]\",\r\n \"Description\": \"List folders the current user has access to. Use --all to list all folders in the tenant (supports filtering and sorting).\"\r\n },\r\n {\r\n \"Name\": \"create [options] \",\r\n \"Description\": \"Create a new folder. Folders scope processes, jobs, and resources with separate permissions. Use --parent to nest inside an existing folder.\"\r\n },\r\n {\r\n \"Name\": \"get [options] \",\r\n \"Description\": \"Get folder details by key (GUID) or path. Returns display name, key (GUID), path, type, and permissions. Use 'folders list' to find folder keys.\"\r\n },\r\n {\r\n \"Name\": \"delete [options] \",\r\n \"Description\": \"Delete a folder by key (GUID) or path. Fails if the folder contains entities or user associations. Move or remove contents before deleting.\"\r\n },\r\n {\r\n \"Name\": \"move [options] \",\r\n \"Description\": \"Move a folder to a new parent in the folder hierarchy. Provide the target parent folder path or key. Use --root to move to the top level.\"\r\n },\r\n {\r\n \"Name\": \"edit [options] \",\r\n \"Description\": \"Edit folder properties (display name, description). Use 'folders get' to see current values before editing.\"\r\n },\r\n {\r\n \"Name\": \"runtimes [options] \",\r\n \"Description\": \"List runtime allocations for a folder. Shows how many runtime slots of each type (Unattended, Attended, NonProduction, etc.) are total, connected, and available in the specified folder. Useful for checking capacity before starting jobs.\"\r\n },\r\n {\r\n \"Name\": \"help [command]\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"22302120-d65a-4f32-bc45-0a0370124f6c\",\r\n \"Name\": \"Debug_InvoiceApproval\",\r\n \"Path\": \"original_email@test.com's workspace/Debug_InvoiceApproval\",\r\n \"Description\": \"\",\r\n \"Type\": \"DebugSolution\",\r\n \"ParentKey\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\"\r\n },\r\n {\r\n \"Key\": \"1f77e545-4388-4b96-a51f-9a0f771dc331\",\r\n \"Name\": \"InvoiceApproval\",\r\n \"Path\": \"original_email@test.com's workspace/InvoiceApproval\",\r\n \"Description\": \"\",\r\n \"Type\": \"Solution\",\r\n \"ParentKey\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\"\r\n },\r\n {\r\n \"Key\": \"445c18c8-3f11-43b6-ad44-6ff83f33af18\",\r\n \"Name\": \"BenefitsEnrollment\",\r\n \"Path\": \"BenefitsEnrollment\",\r\n \"Description\": \"Demo folder for Scenario 2 - Portal That Changed Overnight (DevCon 2026 Troubleshoot Agent)\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"236eb3fa-5689-467a-9025-d57f6a0614a7\",\r\n \"Name\": \"Demo3\",\r\n \"Path\": \"Demo3\",\r\n \"Description\": \"\",\r\n \"Type\": \"Solution\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"7072f52f-6e95-4ef5-b2ab-7e9ee97ea460\",\r\n \"Name\": \"DevCon-Scenario-4\",\r\n \"Path\": \"DevCon-Scenario-4\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"4d992869-8efe-44e1-a363-733606b9b35d\",\r\n \"Name\": \"SupportTriage\",\r\n \"Path\": \"DevCon-Scenario-4/SupportTriage\",\r\n \"Description\": \"\",\r\n \"Type\": \"Solution\",\r\n \"ParentKey\": \"7072f52f-6e95-4ef5-b2ab-7e9ee97ea460\"\r\n },\r\n {\r\n \"Key\": \"62e70ea1-11de-4263-8886-d98e3f20049c\",\r\n \"Name\": \"InvoiceApproval\",\r\n \"Path\": \"InvoiceApproval\",\r\n \"Description\": \"\",\r\n \"Type\": \"Solution\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"3ac175f5-7742-4a0f-a99f-f60678c399e6\",\r\n \"Name\": \"InvoiceApproval 1\",\r\n \"Path\": \"InvoiceApproval 1\",\r\n \"Description\": \"\",\r\n \"Type\": \"Solution\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"8673c3a0-e11a-418b-82d3-864200c6fb97\",\r\n \"Name\": \"InvoiceApprovalTask\",\r\n \"Path\": \"InvoiceApprovalTask\",\r\n \"Description\": \"\",\r\n \"Type\": \"Solution\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"7f984a6b-1138-40fb-b48e-2c3ddf02b8f0\",\r\n \"Name\": \"PurchaseOrderProcessing\",\r\n \"Path\": \"PurchaseOrderProcessing\",\r\n \"Description\": \"Demo folder for PO processing automation\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "--help" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Help\",\r\n \"Data\": {\r\n \"Command\": \"or\",\r\n \"Description\": \"Manage Orchestrator jobs, processes, packages, folders, machines, licenses, users, and permissions. Supports the full automation lifecycle: deploy packages, create processes, execute jobs, provision machines, manage licenses, manage access, and monitor results.\",\r\n \"Usage\": \"uip or [options] [command]\",\r\n \"Arguments\": [],\r\n \"Options\": [\r\n {\r\n \"Flags\": \"-h, --help\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"Subcommands\": [\r\n {\r\n \"Name\": \"attachments\",\r\n \"Description\": \"Manage job attachments. Download files attached to jobs (screenshots, reports, data files). Attachments are tenant-scoped — no folder context needed.\"\r\n },\r\n {\r\n \"Name\": \"audit-logs\",\r\n \"Description\": \"View audit logs for the tenant. Shows who did what and when — tracks changes to users, processes, assets, jobs, and other entities.\"\r\n },\r\n {\r\n \"Name\": \"calendars\",\r\n \"Description\": \"Manage calendars (Non-Working Days). Calendars define excluded dates (holidays) used by time triggers to skip scheduled runs.\"\r\n },\r\n {\r\n \"Name\": \"credential-stores\",\r\n \"Description\": \"Manage Orchestrator credential stores (tenant-scoped secure storage backends for credentials and assets).\"\r\n },\r\n {\r\n \"Name\": \"feeds\",\r\n \"Description\": \"Manage UiPath package feeds. Feeds are repositories that store automation packages (.nupkg). Use the feed ID with 'packages list --feed-id' to browse packages in a specific feed.\"\r\n },\r\n {\r\n \"Name\": \"folders\",\r\n \"Description\": \"Manage UiPath Orchestrator folders. Folders organize and scope resources (processes, jobs, assets, queues) with role-based access. Most commands require --folder-path or --folder-key to target a specific folder.\"\r\n },\r\n {\r\n \"Name\": \"jobs\",\r\n \"Description\": \"Manage UiPath Orchestrator jobs (process executions). Start jobs from processes, monitor execution, check logs and traces.\"\r\n },\r\n {\r\n \"Name\": \"licenses\",\r\n \"Description\": \"Manage UiPath Orchestrator licenses. View license allocation and usage across robot types, list runtime and named-user license assignments, and toggle machine licensing on or off.\"\r\n },\r\n {\r\n \"Name\": \"machines\",\r\n \"Description\": \"Manage UiPath Orchestrator machines (robot host definitions). Machines define the hosts where robots execute automations, along with their license slot allocations. Machines are tenant-scoped but can be assigned to specific folders.\"\r\n },\r\n {\r\n \"Name\": \"packages\",\r\n \"Description\": \"Manage UiPath automation packages (.nupkg) in the Orchestrator feed. Packages are uploaded code artifacts. Use 'processes' to manage runnable process bindings in folders.\"\r\n },\r\n {\r\n \"Name\": \"roles\",\r\n \"Description\": \"Manage Orchestrator roles and permissions (RBAC). Roles group permissions and are assigned to users at the tenant or folder level. Tenant roles apply globally; Folder roles scope access to specific folders. Use 'roles list-roles' to find role keys, 'roles list-permissions' for grantable permissions.\"\r\n },\r\n {\r\n \"Name\": \"sessions\",\r\n \"Description\": \"View and manage Orchestrator sessions (active host connections). Sessions represent live connections between hosts and Orchestrator, including their current state, runtime type, and maintenance status. Use subcommands to list sessions, inspect machine runtimes, toggle debug mode, delete inactive sessions, or set maintenance mode.\"\r\n },\r\n {\r\n \"Name\": \"processes\",\r\n \"Description\": \"Manage UiPath processes (runnable bindings of packages to folders). Processes are what you execute with 'jobs start'. Use 'packages' to manage the uploaded automation code.\"\r\n },\r\n {\r\n \"Name\": \"settings\",\r\n \"Description\": \"Manage tenant settings. Settings are key-value pairs — use 'settings list' to discover keys. Categories and common keys: [General] Abp.Timing.TimeZone (timezone ID, e.g. 'UTC' — use 'settings timezones' to list valid IDs), Abp.Localization.DefaultLanguageName (language code, e.g. 'en', 'fr', 'de'). [Email] Alerts.Email.Enabled (true/false), Abp.Net.Mail.Smtp.Host (hostname), .Port (integer, e.g. 587), .UserName, .Password, .EnableSsl (true/false), .UseDefaultCredentials (true/false), Abp.Net.Mail.DefaultFromAddress (email address). [Robot Security] Auth.Robot.EnforceAuthenticationToken (true/false — force token auth for robots), Auth.UnattendedRobot.ForceClientCredentials (True/False), AttendedRobot.RunDisconnectedHours (integer, 0=disabled, hours a robot can run disconnected). [Deployment] DeploymentUrl (NuGet feed URL), NuGet.Packages.ApiKey (API key string), Deployment.Libraries.Url, NuGet.Activities.ApiKey, Deployment.Processes.AuthenticationType (Secure/apiKey). [Scalability] Scalability.SignalR.Enabled (true/false), Scalability.SignalR.Transport (bitmask: 1=WebSockets, 2=SSE, 4=LongPolling, 7=all), Scalability.Heartbeat.PeriodSeconds (integer, default 30, seconds between robot heartbeats), Scalability.Heartbeat.FailureThreshold (integer, default 4, missed heartbeats before marking not responding). [Triggers] Triggers.DisableWhenFailedCount (integer, default 10, consecutive failures to auto-disable), Triggers.DisableWhenFailingSinceDays (integer, default 1), Triggers.JobsCountStrategy (PERPROCESS/PERTRIGGER), QueueTrigger.ActivationFrequencyInMinutes (integer, default 30), ConnectedTriggers.AutoDisableJobFailuresThreshold (integer, default 5). [Queue] inProgressMaxNumberOfMinutes (integer, default 1440=24h, max time an item stays InProgress). [Jobs] Jobs.TerminatingJobsTimeout (integer, default 1440 minutes=24h, timeout for terminating jobs). Use 'settings execution' for structured execution settings with display names and allowed values.\"\r\n },\r\n {\r\n \"Name\": \"users\",\r\n \"Description\": \"Manage Orchestrator users (tenant-scoped). List, create, edit, and delete users. Assign users to folders with specific roles. Manage tenant-level role assignments. Use 'users list' to find user keys (GUIDs).\"\r\n },\r\n {\r\n \"Name\": \"help [command]\",\r\n \"Description\": \"display help for command\"\r\n }\r\n ],\r\n \"GlobalOptions\": [\r\n {\r\n \"Flags\": \"--output \",\r\n \"Description\": \"Output format (default: json)\"\r\n },\r\n {\r\n \"Flags\": \"--output-filter \",\r\n \"Description\": \"JMESPath expression to filter output\"\r\n },\r\n {\r\n \"Flags\": \"--log-level \",\r\n \"Description\": \"Log level threshold (default: info)\"\r\n },\r\n {\r\n \"Flags\": \"--log-file \",\r\n \"Description\": \"Write logs to file instead of stderr\"\r\n }\r\n ]\r\n }\r\n}" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/task.yaml b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/task.yaml index 296f22dda1..f797486d61 100644 --- a/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/cross-system/rpa-preflight-failure/task.yaml @@ -17,22 +17,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/README.md b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/README.md new file mode 100644 index 0000000000..f1df5170ee --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/README.md @@ -0,0 +1,41 @@ +# Agent Context Grounding - Index Not Found + +Faithful-replay scenario for the `uipath-troubleshoot` skill. It covers an +Agents runtime failure where a context grounding tool span reports +`ContextGroundingIndex not found`. + +## What this exercises + +The trace contains a `contextGroundingTool` span for `SupportKnowledge` with a +missing `support-kb-prod` index. The local agent project has a matching +`SupportAgent/resources/SupportKnowledge/resource.json` that points to +`indexName: support-kb-prod` in `folderPath: Shared/Agents`. The mocked +`uip context-grounding list --folder-path Shared/Agents` response returns no +matching index. + +The agent should diagnose a deleted or never-created Context Grounding index, +then recommend the current lifecycle: + +- inspect the trace with `uip traces spans get` +- inspect the agent-side context `resource.json` +- recreate or relink the index with `uip context-grounding` +- after a resource edit, run `uip agent refresh` and `uip agent validate` +- after successful validation, ask whether the user wants to upload or + publish/deploy the corrected solution + +It must not recommend deprecated agent context-management, standalone publish, +or run-status commands. + +## Mock surface + +| Command | Fixture | +|---|---| +| `traces spans get ` | `trace-context-index-missing.json` | +| `context-grounding list --folder-path Shared/Agents` | `context-grounding-list-empty.json` | +| `context-grounding retrieve` | `context-grounding-retrieve.json` (not found) | +| `agent validate` | `agent-validate.json` | +| repair/upload/publish/deploy commands attempted without approval | `repair-command-denied.json` | + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/README.md deleted file mode 100644 index 569cc9c91a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/README.md +++ /dev/null @@ -1,37 +0,0 @@ -# Agent Context Grounding - Index Not Found - -Faithful-replay scenario for the `uipath-troubleshoot` skill. It covers an -Agents runtime failure where a context grounding tool span reports -`ContextGroundingIndex not found`. - -## What this exercises - -The trace contains a `contextGroundingTool` span for `SupportKnowledge` with a -missing `support-kb-prod` index. The local agent project has a matching -`SupportAgent/resources/SupportKnowledge/resource.json` that points to -`indexName: support-kb-prod` in `folderPath: Shared/Agents`. The mocked -`uip context-grounding list --folder-path Shared/Agents` response returns no -matching index. - -The agent should diagnose a deleted or never-created Context Grounding index, -then recommend the current lifecycle: - -- inspect the trace with `uip traces spans get` -- inspect the agent-side context `resource.json` -- recreate or relink the index with `uip context-grounding` -- after a resource edit, run `uip agent refresh` and `uip agent validate` -- after successful validation, ask whether the user wants to upload or - publish/deploy the corrected solution - -It must not recommend deprecated agent context-management, standalone publish, -or run-status commands. - -## Mock surface - -| Command | Fixture | -|---|---| -| `traces spans get ` | `trace-context-index-missing.json` | -| `context-grounding list --folder-path Shared/Agents` | `context-grounding-list-empty.json` | -| `context-grounding retrieve` | `context-grounding-retrieve.json` (not found) | -| `agent validate` | `agent-validate.json` | -| repair/upload/publish/deploy commands attempted without approval | `repair-command-denied.json` | diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/agent-validate.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/agent-validate.json deleted file mode 100644 index 37e899f461..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/agent-validate.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "AgentValidate", - "Data": { - "Status": "Valid", - "ProjectDir": "process/SupportSolution/SupportAgent" - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/context-grounding-list-empty.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/context-grounding-list-empty.json deleted file mode 100644 index b94b7cd9bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/context-grounding-list-empty.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "ContextGroundingIndexList", - "Data": [ - { - "indexName": "legacy-support-kb", - "folderPath": "Shared/Agents", - "last_ingestion_status": "Successful", - "status": "Active" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/context-grounding-retrieve.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/context-grounding-retrieve.json deleted file mode 100644 index 5411238936..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/context-grounding-retrieve.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ContextGroundingIndexRetrieve", - "Message": "Context Grounding index not found", - "Instructions": "No index named support-kb-prod exists in folder Shared/Agents. Recreate the index or relink the agent resource to an existing active index." -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/manifest.json deleted file mode 100644 index 26e8f68840..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,80 +0,0 @@ -{ - "version": 2, - "_doc": "Agents context grounding index-not-found scenario. First-match-wins; specific lifecycle commands are included to keep command-shape coverage current.", - "rules": [ - { - "match": "traces spans get aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "file": "trace-context-index-missing.json" - }, - { - "match": "context-grounding list --folder-path Shared/Agents", - "file": "context-grounding-list-empty.json" - }, - { - "match": "context-grounding create --index-name support-kb-prod --bucket-source support-kb-source --folder-path Shared/Agents", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "context-grounding ingest --index-name support-kb-prod --folder-path Shared/Agents", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "context-grounding retrieve --index-name support-kb-prod --folder-path Shared/Agents", - "file": "context-grounding-retrieve.json", - "exit_code": 1 - }, - { - "match": "agent refresh process/SupportSolution/SupportAgent", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent validate process/SupportSolution/SupportAgent", - "file": "agent-validate.json" - }, - { - "match": "solution resources refresh", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution upload .", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution pack . ./dist --version 1.0.1", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution publish ./dist/SupportSolution.1.0.1.zip", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution deploy run --name SupportAgent-prod --package-name SupportSolution --package-version 1.0.1 --folder-name Agents --parent-folder-path Shared", - "file": "repair-command-denied.json", - "exit_code": 2 - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for exploratory read-only commands." - }, - "expected_calls": [ - { - "pattern": "traces spans get aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "min": 1, - "description": "Agent inspects trace spans for the failing run." - }, - { - "pattern": "context-grounding list --folder-path Shared/Agents", - "min": 1, - "description": "Agent verifies whether the configured index exists in the deployment folder." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/repair-command-denied.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/repair-command-denied.json deleted file mode 100644 index 3bf034b0ed..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/repair-command-denied.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ScenarioRepairCommandDenied", - "Message": "This scenario is diagnose-only.", - "Instructions": "Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action." -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/trace-context-index-missing.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/trace-context-index-missing.json deleted file mode 100644 index e8eca04527..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/m/r/trace-context-index-missing.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "spanId": "ctx-0001", - "traceId": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "name": "SupportKnowledge", - "spanType": "contextGroundingTool", - "SPANTYPE": "contextGroundingTool", - "attributes": { - "error": "Unexpected Error Details: An unexpected error occurred during agent execution, please try again later or contact your Administrator. Error Details: ContextGroundingIndex not found Code: AGENT_RUNTIME.UNEXPECTED_ERROR", - "indexName": "support-kb-prod", - "folderPath": "Shared/Agents", - "contextResourceName": "SupportKnowledge" - }, - "ATTRIBUTES": { - "error": "Unexpected Error Details: An unexpected error occurred during agent execution, please try again later or contact your Administrator. Error Details: ContextGroundingIndex not found Code: AGENT_RUNTIME.UNEXPECTED_ERROR", - "indexName": "support-kb-prod", - "folderPath": "Shared/Agents", - "contextResourceName": "SupportKnowledge" - } - }, - { - "spanId": "run-0001", - "traceId": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "name": "agentRun", - "spanType": "agentRun", - "attributes": { - "state": "Faulted" - } - } - ], - "spans": [ - { - "name": "SupportKnowledge", - "spanType": "contextGroundingTool", - "attributes": { - "error": "ContextGroundingIndex not found Code: AGENT_RUNTIME.UNEXPECTED_ERROR", - "indexName": "support-kb-prod", - "folderPath": "Shared/Agents" - } - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..1563a65abd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/data/uip-fixture.json @@ -0,0 +1,167 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "solution", + "deploy", + "run", + "--name", + "SupportAgent-prod", + "--package-name", + "SupportSolution", + "--package-version", + "1.0.1", + "--folder-name", + "Agents", + "--parent-folder-path", + "Shared" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "context-grounding", + "create", + "--index-name", + "support-kb-prod", + "--bucket-source", + "support-kb-source", + "--folder-path", + "Shared/Agents" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "context-grounding", + "ingest", + "--index-name", + "support-kb-prod", + "--folder-path", + "Shared/Agents" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "context-grounding", + "retrieve", + "--index-name", + "support-kb-prod", + "--folder-path", + "Shared/Agents" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ContextGroundingIndexRetrieve\",\r\n \"Message\": \"Context Grounding index not found\",\r\n \"Instructions\": \"No index named support-kb-prod exists in folder Shared/Agents. Recreate the index or relink the agent resource to an existing active index.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "pack", + ".", + "./dist", + "--version", + "1.0.1" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"spanId\": \"ctx-0001\",\r\n \"traceId\": \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\r\n \"name\": \"SupportKnowledge\",\r\n \"spanType\": \"contextGroundingTool\",\r\n \"SPANTYPE\": \"contextGroundingTool\",\r\n \"attributes\": {\r\n \"error\": \"Unexpected Error Details: An unexpected error occurred during agent execution, please try again later or contact your Administrator. Error Details: ContextGroundingIndex not found Code: AGENT_RUNTIME.UNEXPECTED_ERROR\",\r\n \"indexName\": \"support-kb-prod\",\r\n \"folderPath\": \"Shared/Agents\",\r\n \"contextResourceName\": \"SupportKnowledge\"\r\n },\r\n \"ATTRIBUTES\": {\r\n \"error\": \"Unexpected Error Details: An unexpected error occurred during agent execution, please try again later or contact your Administrator. Error Details: ContextGroundingIndex not found Code: AGENT_RUNTIME.UNEXPECTED_ERROR\",\r\n \"indexName\": \"support-kb-prod\",\r\n \"folderPath\": \"Shared/Agents\",\r\n \"contextResourceName\": \"SupportKnowledge\"\r\n }\r\n },\r\n {\r\n \"spanId\": \"run-0001\",\r\n \"traceId\": \"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\r\n \"name\": \"agentRun\",\r\n \"spanType\": \"agentRun\",\r\n \"attributes\": {\r\n \"state\": \"Faulted\"\r\n }\r\n }\r\n ],\r\n \"spans\": [\r\n {\r\n \"name\": \"SupportKnowledge\",\r\n \"spanType\": \"contextGroundingTool\",\r\n \"attributes\": {\r\n \"error\": \"ContextGroundingIndex not found Code: AGENT_RUNTIME.UNEXPECTED_ERROR\",\r\n \"indexName\": \"support-kb-prod\",\r\n \"folderPath\": \"Shared/Agents\"\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "context-grounding", + "list", + "--folder-path", + "Shared/Agents" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ContextGroundingIndexList\",\r\n \"Data\": [\r\n {\r\n \"indexName\": \"legacy-support-kb\",\r\n \"folderPath\": \"Shared/Agents\",\r\n \"last_ingestion_status\": \"Successful\",\r\n \"status\": \"Active\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "agent", + "refresh", + "process/SupportSolution/SupportAgent" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "validate", + "process/SupportSolution/SupportAgent" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AgentValidate\",\r\n \"Data\": {\r\n \"Status\": \"Valid\",\r\n \"ProjectDir\": \"process/SupportSolution/SupportAgent\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "solution", + "resources", + "refresh" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "upload", + "." + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "publish", + "./dist/SupportSolution.1.0.1.zip" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run mutating repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "traces spans get aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "min": 1, + "description": "Agent inspects trace spans for the failing run." + }, + { + "pattern": "context-grounding list --folder-path Shared/Agents", + "min": 1, + "description": "Agent verifies whether the configured index exists in the deployment folder." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/task.yaml index ffa66715f6..26439d57f1 100644 --- a/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/agents/context-grounding-index-not-found/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/README.md b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/README.md new file mode 100644 index 0000000000..a1743dd7cc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/README.md @@ -0,0 +1,27 @@ +# Agent Input Schema Validation Failure + +Faithful-replay scenario for the `uipath-troubleshoot` skill. It covers an +Agents invocation where the payload omits a required input field. + +## What this exercises + +The trace contains an `agentRun` span with: + +`Input validation failed Details: Data failed json schema validation ... customerEmail Field required` + +The local `IntakeAgent/agent.json` declares `customerEmail` as a required +string input. The fixture includes denied `uip agent debug` rules for command +shape coverage; the agent should present the corrected debug command instead of +running a live debug session during diagnosis. + +The agent should diagnose a bad caller payload, show a corrected `uip agent +debug --inputs` command, and only suggest editing `agent.json` plus +refresh/validate when the schema itself needs changing. After validation, it +should ask whether the user wants to upload or publish/deploy the correction. + +It must not use deprecated agent run, input-management, or standalone publish +commands. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/README.md deleted file mode 100644 index 453bfdec63..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/README.md +++ /dev/null @@ -1,23 +0,0 @@ -# Agent Input Schema Validation Failure - -Faithful-replay scenario for the `uipath-troubleshoot` skill. It covers an -Agents invocation where the payload omits a required input field. - -## What this exercises - -The trace contains an `agentRun` span with: - -`Input validation failed Details: Data failed json schema validation ... customerEmail Field required` - -The local `IntakeAgent/agent.json` declares `customerEmail` as a required -string input. The fixture includes denied `uip agent debug` rules for command -shape coverage; the agent should present the corrected debug command instead of -running a live debug session during diagnosis. - -The agent should diagnose a bad caller payload, show a corrected `uip agent -debug --inputs` command, and only suggest editing `agent.json` plus -refresh/validate when the schema itself needs changing. After validation, it -should ask whether the user wants to upload or publish/deploy the correction. - -It must not use deprecated agent run, input-management, or standalone publish -commands. diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/agent-validate.json b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/agent-validate.json deleted file mode 100644 index 2a15e0823c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/agent-validate.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "AgentValidate", - "Data": { - "Status": "Valid", - "ProjectDir": "process/IntakeSolution/IntakeAgent" - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/debug-command-denied.json b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/debug-command-denied.json deleted file mode 100644 index 8d7422820d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/debug-command-denied.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ScenarioDebugCommandDenied", - "Message": "This scenario is diagnose-only.", - "Instructions": "Do not run uip agent debug without explicit user approval. Provide the corrected debug command for the user to run instead." -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/manifest.json deleted file mode 100644 index 96f9cba78e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/manifest.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "version": 2, - "_doc": "Agents input schema validation scenario. Covers trace-first diagnosis and command-shape coverage for uip agent debug without allowing live debug execution in diagnose-only mode.", - "rules": [ - { - "match": "traces spans get bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", - "file": "trace-input-schema.json" - }, - { - "match": "agent debug process/IntakeSolution/IntakeAgent --inputs {\"request\":\"create onboarding case\"}", - "file": "debug-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent debug process/IntakeSolution/IntakeAgent --inputs {\"request\":\"create onboarding case\",\"customerEmail\":\"customer@example.com\"}", - "file": "debug-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent refresh process/IntakeSolution/IntakeAgent", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent validate process/IntakeSolution/IntakeAgent", - "file": "agent-validate.json" - }, - { - "match": "solution upload .", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution pack . ./dist --version 1.0.1", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution publish ./dist/IntakeSolution.1.0.1.zip", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution deploy run --name IntakeAgent-prod --package-name IntakeSolution --package-version 1.0.1 --folder-name Agents --parent-folder-path Shared", - "file": "repair-command-denied.json", - "exit_code": 2 - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for exploratory read-only commands." - }, - "expected_calls": [ - { - "pattern": "traces spans get bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", - "min": 1, - "description": "Agent inspects the trace for the agentRun validation error." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/repair-command-denied.json b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/repair-command-denied.json deleted file mode 100644 index 3ff18ebb49..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/repair-command-denied.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ScenarioRepairCommandDenied", - "Message": "This scenario is diagnose-only.", - "Instructions": "Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action." -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/trace-input-schema.json b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/trace-input-schema.json deleted file mode 100644 index 17bbb07853..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/m/r/trace-input-schema.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "spanId": "run-0002", - "traceId": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", - "name": "agentRun", - "spanType": "agentRun", - "SPANTYPE": "agentRun", - "attributes": { - "error": "Input validation failed Details: Data failed json schema validation: 1 validation error for DynamicType_0 BatchJson customerEmail Field required [type=missing, input_value={'request': 'create onboarding case'}, input_type=dict]", - "agentName": "IntakeAgent" - }, - "ATTRIBUTES": { - "error": "Input validation failed Details: Data failed json schema validation: 1 validation error for DynamicType_0 BatchJson customerEmail Field required [type=missing, input_value={'request': 'create onboarding case'}, input_type=dict]", - "agentName": "IntakeAgent" - } - } - ], - "spans": [ - { - "name": "agentRun", - "spanType": "agentRun", - "attributes": { - "error": "Input validation failed Details: Data failed json schema validation: 1 validation error for DynamicType_0 BatchJson customerEmail Field required" - } - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/uip-fixture.json new file mode 100644 index 0000000000..bccf00a600 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "solution", + "deploy", + "run", + "--name", + "IntakeAgent-prod", + "--package-name", + "IntakeSolution", + "--package-version", + "1.0.1", + "--folder-name", + "Agents", + "--parent-folder-path", + "Shared" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "pack", + ".", + "./dist", + "--version", + "1.0.1" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "debug", + "process/IntakeSolution/IntakeAgent", + "--inputs", + "{request:create onboarding case}" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioDebugCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run uip agent debug without explicit user approval. Provide the corrected debug command for the user to run instead.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "debug", + "process/IntakeSolution/IntakeAgent", + "--inputs", + "{request:create onboarding case,customerEmail:customer@example.com}" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioDebugCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run uip agent debug without explicit user approval. Provide the corrected debug command for the user to run instead.\"\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"spanId\": \"run-0002\",\r\n \"traceId\": \"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\r\n \"name\": \"agentRun\",\r\n \"spanType\": \"agentRun\",\r\n \"SPANTYPE\": \"agentRun\",\r\n \"attributes\": {\r\n \"error\": \"Input validation failed Details: Data failed json schema validation: 1 validation error for DynamicType_0 BatchJson customerEmail Field required [type=missing, input_value={'request': 'create onboarding case'}, input_type=dict]\",\r\n \"agentName\": \"IntakeAgent\"\r\n },\r\n \"ATTRIBUTES\": {\r\n \"error\": \"Input validation failed Details: Data failed json schema validation: 1 validation error for DynamicType_0 BatchJson customerEmail Field required [type=missing, input_value={'request': 'create onboarding case'}, input_type=dict]\",\r\n \"agentName\": \"IntakeAgent\"\r\n }\r\n }\r\n ],\r\n \"spans\": [\r\n {\r\n \"name\": \"agentRun\",\r\n \"spanType\": \"agentRun\",\r\n \"attributes\": {\r\n \"error\": \"Input validation failed Details: Data failed json schema validation: 1 validation error for DynamicType_0 BatchJson customerEmail Field required\"\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "agent", + "refresh", + "process/IntakeSolution/IntakeAgent" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "validate", + "process/IntakeSolution/IntakeAgent" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AgentValidate\",\r\n \"Data\": {\r\n \"Status\": \"Valid\",\r\n \"ProjectDir\": \"process/IntakeSolution/IntakeAgent\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "solution", + "upload", + "." + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "publish", + "./dist/IntakeSolution.1.0.1.zip" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "traces spans get bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "min": 1, + "description": "Agent inspects the trace for the agentRun validation error." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/task.yaml b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/task.yaml index 93872ab5a3..f9e080d2b5 100644 --- a/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/agents/input-schema-validation-failure/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/README.md b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/README.md new file mode 100644 index 0000000000..7656e012a9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/README.md @@ -0,0 +1,28 @@ +# Agent LLM Insufficient Information + +Faithful-replay scenario for the `uipath-troubleshoot` skill. It covers an +Agents runtime failure where a completion span returns a JSON error whose +`detail` says the recipient is missing. + +## What this exercises + +The trace contains a failing `completion` span with: + +`{"detail":"Insufficient information to send the email: recipient is missing"}` + +The local agent prompt says only "Send the requested email update" and the +input schema has a single `task` field. The agent should identify that the +system prompt and/or programmatic input schema does not force the caller to +provide a recipient. + +The expected fix is to edit `agent.json` directly: update +`messages[0].content`, rebuild `contentTokens`, add an input field such as +`recipient` when programmatic calls need it, then run refresh/validate. After +validation, the agent should ask whether the user wants to upload or +publish/deploy the correction. Local reproduction should be presented as a +`uip agent debug` command and run only after explicit user approval, because +debug uploads and executes the agent. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/README.md deleted file mode 100644 index 4bdca395b6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/README.md +++ /dev/null @@ -1,24 +0,0 @@ -# Agent LLM Insufficient Information - -Faithful-replay scenario for the `uipath-troubleshoot` skill. It covers an -Agents runtime failure where a completion span returns a JSON error whose -`detail` says the recipient is missing. - -## What this exercises - -The trace contains a failing `completion` span with: - -`{"detail":"Insufficient information to send the email: recipient is missing"}` - -The local agent prompt says only "Send the requested email update" and the -input schema has a single `task` field. The agent should identify that the -system prompt and/or programmatic input schema does not force the caller to -provide a recipient. - -The expected fix is to edit `agent.json` directly: update -`messages[0].content`, rebuild `contentTokens`, add an input field such as -`recipient` when programmatic calls need it, then run refresh/validate. After -validation, the agent should ask whether the user wants to upload or -publish/deploy the correction. Local reproduction should be presented as a -`uip agent debug` command and run only after explicit user approval, because -debug uploads and executes the agent. diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/agent-validate.json b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/agent-validate.json deleted file mode 100644 index 83c64ddc96..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/agent-validate.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "AgentValidate", - "Data": { - "Status": "Valid", - "ProjectDir": "process/MailerSolution/MailerAgent" - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/debug-command-denied.json b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/debug-command-denied.json deleted file mode 100644 index 8d7422820d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/debug-command-denied.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ScenarioDebugCommandDenied", - "Message": "This scenario is diagnose-only.", - "Instructions": "Do not run uip agent debug without explicit user approval. Provide the corrected debug command for the user to run instead." -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/manifest.json deleted file mode 100644 index 1b1fe9e658..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/manifest.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "version": 2, - "_doc": "Agents insufficient-information scenario. Covers trace parsing and command-shape coverage for uip agent debug without allowing live debug execution in diagnose-only mode.", - "rules": [ - { - "match": "traces spans get cccccccccccccccccccccccccccccccc", - "file": "trace-insufficient-information.json" - }, - { - "match": "agent debug process/MailerSolution/MailerAgent --inputs {\"task\":\"send update\"}", - "file": "debug-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent debug process/MailerSolution/MailerAgent --inputs {\"task\":\"send update\",\"recipient\":\"customer@example.com\"}", - "file": "debug-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent refresh process/MailerSolution/MailerAgent", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "agent validate process/MailerSolution/MailerAgent", - "file": "agent-validate.json" - }, - { - "match": "solution upload .", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution pack . ./dist --version 1.0.1", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution publish ./dist/MailerSolution.1.0.1.zip", - "file": "repair-command-denied.json", - "exit_code": 2 - }, - { - "match": "solution deploy run --name MailerAgent-prod --package-name MailerSolution --package-version 1.0.1 --folder-name Agents --parent-folder-path Shared", - "file": "repair-command-denied.json", - "exit_code": 2 - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback for exploratory read-only commands." - }, - "expected_calls": [ - { - "pattern": "traces spans get cccccccccccccccccccccccccccccccc", - "min": 1, - "description": "Agent inspects trace spans for the completion error." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/repair-command-denied.json b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/repair-command-denied.json deleted file mode 100644 index 3ff18ebb49..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/repair-command-denied.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ScenarioRepairCommandDenied", - "Message": "This scenario is diagnose-only.", - "Instructions": "Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action." -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/trace-insufficient-information.json b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/trace-insufficient-information.json deleted file mode 100644 index 934f154b57..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/m/r/trace-insufficient-information.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "spanId": "completion-0003", - "traceId": "cccccccccccccccccccccccccccccccc", - "name": "llm.completion", - "spanType": "completion", - "SPANTYPE": "completion", - "attributes": { - "error": "{\"detail\":\"Insufficient information to send the email: recipient is missing\"}", - "model": "gpt-5.4" - }, - "ATTRIBUTES": { - "error": "{\"detail\":\"Insufficient information to send the email: recipient is missing\"}", - "model": "gpt-5.4" - } - } - ], - "spans": [ - { - "name": "llm.completion", - "spanType": "completion", - "attributes": { - "error": "{\"detail\":\"Insufficient information to send the email: recipient is missing\"}" - } - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/uip-fixture.json new file mode 100644 index 0000000000..1e7bc2c66a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/data/uip-fixture.json @@ -0,0 +1,124 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "solution", + "deploy", + "run", + "--name", + "MailerAgent-prod", + "--package-name", + "MailerSolution", + "--package-version", + "1.0.1", + "--folder-name", + "Agents", + "--parent-folder-path", + "Shared" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "pack", + ".", + "./dist", + "--version", + "1.0.1" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "debug", + "process/MailerSolution/MailerAgent", + "--inputs", + "{task:send update}" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioDebugCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run uip agent debug without explicit user approval. Provide the corrected debug command for the user to run instead.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "debug", + "process/MailerSolution/MailerAgent", + "--inputs", + "{task:send update,recipient:customer@example.com}" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioDebugCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run uip agent debug without explicit user approval. Provide the corrected debug command for the user to run instead.\"\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "cccccccccccccccccccccccccccccccc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"spanId\": \"completion-0003\",\r\n \"traceId\": \"cccccccccccccccccccccccccccccccc\",\r\n \"name\": \"llm.completion\",\r\n \"spanType\": \"completion\",\r\n \"SPANTYPE\": \"completion\",\r\n \"attributes\": {\r\n \"error\": \"{\\\"detail\\\":\\\"Insufficient information to send the email: recipient is missing\\\"}\",\r\n \"model\": \"gpt-5.4\"\r\n },\r\n \"ATTRIBUTES\": {\r\n \"error\": \"{\\\"detail\\\":\\\"Insufficient information to send the email: recipient is missing\\\"}\",\r\n \"model\": \"gpt-5.4\"\r\n }\r\n }\r\n ],\r\n \"spans\": [\r\n {\r\n \"name\": \"llm.completion\",\r\n \"spanType\": \"completion\",\r\n \"attributes\": {\r\n \"error\": \"{\\\"detail\\\":\\\"Insufficient information to send the email: recipient is missing\\\"}\"\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "agent", + "refresh", + "process/MailerSolution/MailerAgent" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "agent", + "validate", + "process/MailerSolution/MailerAgent" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"AgentValidate\",\r\n \"Data\": {\r\n \"Status\": \"Valid\",\r\n \"ProjectDir\": \"process/MailerSolution/MailerAgent\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "solution", + "upload", + "." + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + }, + { + "argv": [ + "solution", + "publish", + "./dist/MailerSolution.1.0.1.zip" + ], + "match_mode": "subset", + "exit_code": 2, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ScenarioRepairCommandDenied\",\r\n \"Message\": \"This scenario is diagnose-only.\",\r\n \"Instructions\": \"Do not run repair, upload, publish, or deploy commands in this replay. Explain the fix and validation steps, then ask before any upload, publish, or deployment action.\"\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "traces spans get cccccccccccccccccccccccccccccccc", + "min": 1, + "description": "Agent inspects trace spans for the completion error." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/task.yaml b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/task.yaml index 593570cab8..8f314e7053 100644 --- a/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/agents/llm-insufficient-information/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/README.md new file mode 100644 index 0000000000..6279d64db8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/README.md @@ -0,0 +1,36 @@ +# Connector Activity — AggregateException (unwrap to inner cause) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the +`System.AggregateException` case of TROUB-139 (`ConnectorActivity`). + +## What this exercises + +A `ConnectorActivity` ("Salesforce: Create Record") faults with +`System.AggregateException` whose inner exception is a `GeneralException` +**DAP-GE-3000** with detail **Bad Gateway** (a transient 502 during connection +resolution). The agent must **unwrap `InnerExceptions[0]`** — the aggregate is +only the async wrapper — and conclude a transient platform error, recommending a +retry. The agent must NOT treat `AggregateException` itself as the root cause. + +Signature mined verbatim from the failed-job telemetry CSV. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, AggregateException + inner GeneralException) | +| `or jobs logs --level Error` | `or-jobs-logs.json` | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence +(the inner exception is in the Info / log). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/README.md deleted file mode 100644 index 7c52f4af38..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# Connector Activity — AggregateException (unwrap to inner cause) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the -`System.AggregateException` case of TROUB-139 (`ConnectorActivity`). - -## What this exercises - -A `ConnectorActivity` ("Salesforce: Create Record") faults with -`System.AggregateException` whose inner exception is a `GeneralException` -**DAP-GE-3000** with detail **Bad Gateway** (a transient 502 during connection -resolution). The agent must **unwrap `InnerExceptions[0]`** — the aggregate is -only the async wrapper — and conclude a transient platform error, recommending a -retry. The agent must NOT treat `AggregateException` itself as the root cause. - -Signature mined verbatim from the failed-job telemetry CSV. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, AggregateException + inner GeneralException) | -| `or jobs logs --level Error` | `or-jobs-logs.json` | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence -(the inner exception is in the Info / log). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/ae49173dee.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/ae49173dee.json deleted file mode 100644 index 55b638eded..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/ae49173dee.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", - "State": "Faulted", - "StartTime": "2026-06-20T11:08:55.330Z", - "EndTime": "2026-06-20T11:08:58.612Z", - "ReleaseName": "LeadSync", - "Source": "Agent", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 5052240 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/cf28f5bbd6.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/cf28f5bbd6.json deleted file mode 100644 index 563def5241..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/cf28f5bbd6.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", - "StartTime": "2026-06-20T11:08:55.330Z", - "EndTime": "2026-06-20T11:08:58.612Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", - "Info": "One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\r\n\r\nin LeadSync.xaml\r\n at ConnectorActivity \"Salesforce: Create Record\"\r\n at Sequence \"LeadSync\"\r\n\r\nSystem.AggregateException: One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\r\n ---> UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n --- End of inner exception stack trace ---\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T11:08:55.330Z", - "ReleaseName": "LeadSync", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "ReleaseVersionId": 30156, - "EntryPointPath": "LeadSync.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-06-20T11:08:58.612Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d3b8a4f29e5a4c7db2e38f1c6d4a0b59", - "Id": 5052240 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/e0bb162c9d.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/e0bb162c9d.json deleted file mode 100644 index 0d3ba59064..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/e0bb162c9d.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T11:08:58.470Z", - "Message": "Salesforce: Create Record: One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\r\nin LeadSync.xaml\r\n at ConnectorActivity \"Salesforce: Create Record\"\r\n at Sequence \"LeadSync\"\r\n\r\n--- .NET stack trace ---\r\nSystem.AggregateException: One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\r\n ---> UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n --- End of inner exception stack trace ---" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/manifest.json deleted file mode 100644 index fba6421a4e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/m/r/manifest.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: ConnectorActivity faults with System.AggregateException wrapping an inner GeneralException DAP-GE-3000 (Bad Gateway during connection resolution — transient). The agent must unwrap InnerExceptions[0]. Signature mined from telemetry. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "ae49173dee.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "ae49173dee.json" - }, - { - "match": "or jobs get d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", - "file": "cf28f5bbd6.json" - }, - { - "match": "or jobs logs d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", - "file": "e0bb162c9d.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", - "min": 1, - "description": "Agent fetches the faulted job to read the AggregateException and its inner GeneralException DAP-GE-3000 (Bad Gateway)." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/uip-fixture.json new file mode 100644 index 0000000000..a4aa2f08fb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/data/uip-fixture.json @@ -0,0 +1,79 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:55.330Z\",\r\n \"EndTime\": \"2026-06-20T11:08:58.612Z\",\r\n \"ReleaseName\": \"LeadSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5052240\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T11:08:55.330Z\",\r\n \"EndTime\": \"2026-06-20T11:08:58.612Z\",\r\n \"ReleaseName\": \"LeadSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5052240\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59\",\r\n \"StartTime\": \"2026-06-20T11:08:55.330Z\",\r\n \"EndTime\": \"2026-06-20T11:08:58.612Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59\",\r\n \"Info\": \"One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\\r\\n\\r\\nin LeadSync.xaml\\r\\n at ConnectorActivity \\\"Salesforce: Create Record\\\"\\r\\n at Sequence \\\"LeadSync\\\"\\r\\n\\r\\nSystem.AggregateException: One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\\r\\n ---> UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n --- End of inner exception stack trace ---\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T11:08:55.330Z\",\r\n \"ReleaseName\": \"LeadSync\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"ReleaseVersionId\": 30156,\r\n \"EntryPointPath\": \"LeadSync.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-06-20T11:08:58.612Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d3b8a4f29e5a4c7db2e38f1c6d4a0b59\",\r\n \"Id\": 5052240\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T11:08:58.470Z\",\r\n \"Message\": \"Salesforce: Create Record: One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\\r\\nin LeadSync.xaml\\r\\n at ConnectorActivity \\\"Salesforce: Create Record\\\"\\r\\n at Sequence \\\"LeadSync\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.AggregateException: One or more errors occurred. (Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.)\\r\\n ---> UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Bad Gateway Error code: DAP-GE-3000.\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n --- End of inner exception stack trace ---\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get d3b8a4f2-9e5a-4c7d-b2e3-8f1c6d4a0b59", + "min": 1, + "description": "Agent fetches the faulted job to read the AggregateException and its inner GeneralException DAP-GE-3000 (Bad Gateway)." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/task.yaml index 9b9a180324..576da4d8a5 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-aggregate/task.yaml @@ -18,21 +18,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/README.md new file mode 100644 index 0000000000..491efa4ae0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/README.md @@ -0,0 +1,43 @@ +# Connector Activity — GeneralException DAP-GE-3005 (connection disabled) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers TROUB-141 +(`ConnectorHttpActivity`) and the headline `GeneralException` case of TROUB-139. + +## What this exercises + +An unattended job faults the moment a `ConnectorHttpActivity` ("Salesforce: HTTP +Request") tries to resolve its Integration Service connection. The connection +exists and is bound, but is in a **Disabled** state, so the IS runtime throws +`GeneralException` with error code **DAP-GE-3005** before any external API call. + +The agent must: +1. Trigger the skill, read the faulted job, and key off the verbatim + `DAP-GE-3005` / "Connection is disabled" signature. +2. Conclude the connection is *disabled* (not missing / no-access / expired) and + recommend re-enabling (and re-authenticating if it auto-disabled). + +Signatures (message text, error code, stack frames) were mined verbatim from the +failed-job telemetry CSV — not invented. + +## Mock surface + + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, Info = DAP-GE-3005) | +| `or jobs logs --level Error` | `or-jobs-logs.json` | +| `is connections ping ` | `is-connections-ping.json` (disabled, exit 1) | +| `is connections list` | `is-connections-list.json` (State: Disabled) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence +alone; the simulator tells the agent source is unavailable and to proceed. + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/README.md deleted file mode 100644 index 67e654cc07..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# Connector Activity — GeneralException DAP-GE-3005 (connection disabled) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers TROUB-141 -(`ConnectorHttpActivity`) and the headline `GeneralException` case of TROUB-139. - -## What this exercises - -An unattended job faults the moment a `ConnectorHttpActivity` ("Salesforce: HTTP -Request") tries to resolve its Integration Service connection. The connection -exists and is bound, but is in a **Disabled** state, so the IS runtime throws -`GeneralException` with error code **DAP-GE-3005** before any external API call. - -The agent must: -1. Trigger the skill, read the faulted job, and key off the verbatim - `DAP-GE-3005` / "Connection is disabled" signature. -2. Conclude the connection is *disabled* (not missing / no-access / expired) and - recommend re-enabling (and re-authenticating if it auto-disabled). - -Signatures (message text, error code, stack frames) were mined verbatim from the -failed-job telemetry CSV — not invented. - -## Mock surface - -CLI is mocked via the shared manifest dispatcher (`../_shared/mock_template`): - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, Info = DAP-GE-3005) | -| `or jobs logs --level Error` | `or-jobs-logs.json` | -| `is connections ping ` | `is-connections-ping.json` (disabled, exit 1) | -| `is connections list` | `is-connections-list.json` (State: Disabled) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence -alone; the simulator tells the agent source is unavailable and to proceed. - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/af051405d6.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/af051405d6.json deleted file mode 100644 index be8287c008..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/af051405d6.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", - "State": "Faulted", - "StartTime": "2026-06-18T09:14:51.120Z", - "EndTime": "2026-06-18T09:14:52.004Z", - "ReleaseName": "OrderSync", - "Source": "Agent", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 5021143 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/b80769b46b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/b80769b46b.json deleted file mode 100644 index 8a11e283ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/b80769b46b.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-18T09:14:51.880Z", - "Message": "Salesforce: HTTP Request: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\r\nin OrderSync.xaml\r\n at ConnectorHttpActivity \"Salesforce: HTTP Request\"\r\n at Sequence \"OrderSync\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.ResolveConnectionAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorHttpActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/c9926c29b1.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/c9926c29b1.json deleted file mode 100644 index 4eb6aab2f3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/c9926c29b1.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "Result": "Failure", - "Code": "ConnectionPing", - "Message": "Connection 'c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40' is disabled.", - "Data": { - "Id": "c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40", - "IsActive": false, - "State": "Disabled" - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/d7498cee88.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/d7498cee88.json deleted file mode 100644 index 97c0512705..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/d7498cee88.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", - "StartTime": "2026-06-18T09:14:51.120Z", - "EndTime": "2026-06-18T09:14:52.004Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", - "Info": "Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\r\n\r\nin OrderSync.xaml\r\n at ConnectorHttpActivity \"Salesforce: HTTP Request\"\r\n at Sequence \"OrderSync\"\r\n\r\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.ResolveConnectionAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorHttpActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-18T09:14:51.120Z", - "ReleaseName": "OrderSync", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "ReleaseVersionId": 10042, - "EntryPointPath": "OrderSync.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-06-18T09:14:52.004Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3f8a1c7e9b2d4e6a8c115d7e9f0a1b2c", - "Id": 5021143 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/ec89315fd6.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/ec89315fd6.json deleted file mode 100644 index 996c3c6240..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/ec89315fd6.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40", - "Name": "Salesforce - Production", - "ConnectorKey": "uipath-salesforce", - "ConnectorName": "Salesforce", - "IsActive": false, - "State": "Disabled", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Owner": "original_email@test.com" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/manifest.json deleted file mode 100644 index e525aaa5e1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/m/r/manifest.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: ConnectorHttpActivity faults with GeneralException DAP-GE-3005 (connection disabled). Signature mined verbatim from failed-job telemetry. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "af051405d6.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "af051405d6.json" - }, - { - "match": "or jobs get 3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", - "file": "d7498cee88.json" - }, - { - "match": "or jobs logs 3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", - "file": "b80769b46b.json" - }, - { - "match": "is connections ping c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40", - "file": "c9926c29b1.json", - "exit_code": 1 - }, - { - "match": "is connections list", - "file": "ec89315fd6.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get 3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", - "min": 1, - "description": "Agent fetches the faulted job to read the GeneralException DAP-GE-3005 'Connection is disabled' signature and the ConnectorHttpActivity stack frame." - }, - { - "pattern": "is connections", - "min": 0, - "description": "OPTIONAL — agent may ping/list the connection to corroborate the disabled state, but the job Info already names the cause. min:0 means tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/uip-fixture.json new file mode 100644 index 0000000000..881494dd04 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/data/uip-fixture.json @@ -0,0 +1,105 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-18T09:14:51.120Z\",\r\n \"EndTime\": \"2026-06-18T09:14:52.004Z\",\r\n \"ReleaseName\": \"OrderSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5021143\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-18T09:14:51.120Z\",\r\n \"EndTime\": \"2026-06-18T09:14:52.004Z\",\r\n \"ReleaseName\": \"OrderSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5021143\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c\",\r\n \"StartTime\": \"2026-06-18T09:14:51.120Z\",\r\n \"EndTime\": \"2026-06-18T09:14:52.004Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c\",\r\n \"Info\": \"Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\r\\n\\r\\nin OrderSync.xaml\\r\\n at ConnectorHttpActivity \\\"Salesforce: HTTP Request\\\"\\r\\n at Sequence \\\"OrderSync\\\"\\r\\n\\r\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.ResolveConnectionAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorHttpActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-18T09:14:51.120Z\",\r\n \"ReleaseName\": \"OrderSync\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"ReleaseVersionId\": 10042,\r\n \"EntryPointPath\": \"OrderSync.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-06-18T09:14:52.004Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3f8a1c7e9b2d4e6a8c115d7e9f0a1b2c\",\r\n \"Id\": 5021143\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-18T09:14:51.880Z\",\r\n \"Message\": \"Salesforce: HTTP Request: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\r\\nin OrderSync.xaml\\r\\n at ConnectorHttpActivity \\\"Salesforce: HTTP Request\\\"\\r\\n at Sequence \\\"OrderSync\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.ResolveConnectionAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorHttpActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ConnectionPing\",\r\n \"Message\": \"Connection 'c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40' is disabled.\",\r\n \"Data\": {\r\n \"Id\": \"c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40\",\r\n \"IsActive\": false,\r\n \"State\": \"Disabled\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"c4d8f0a2-6b1e-4d93-a7c5-8f201b3e9d40\",\r\n \"Name\": \"Salesforce - Production\",\r\n \"ConnectorKey\": \"uipath-salesforce\",\r\n \"ConnectorName\": \"Salesforce\",\r\n \"IsActive\": false,\r\n \"State\": \"Disabled\",\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Owner\": \"original_email@test.com\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get 3f8a1c7e-9b2d-4e6a-8c11-5d7e9f0a1b2c", + "min": 1, + "description": "Agent fetches the faulted job to read the GeneralException DAP-GE-3005 'Connection is disabled' signature and the ConnectorHttpActivity stack frame." + }, + { + "pattern": "is connections", + "min": 0, + "description": "OPTIONAL — agent may ping/list the connection to corroborate the disabled state, but the job Info already names the cause. min:0 means tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/task.yaml index 5c344c8e0e..0c92728ff6 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-disabled/task.yaml @@ -17,21 +17,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/README.md new file mode 100644 index 0000000000..3b00577fe0 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/README.md @@ -0,0 +1,44 @@ +# Connector Activity — GeneralException DAP-GE-3000 (robot lacks Connections.View) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the +high-volume `GeneralException DAP-GE-3000` case of TROUB-139 / TROUB-141 +(`ConnectorActivity` / `ConnectorHttpActivity`). + +## What this exercises + +A `ConnectorActivity` ("Google Drive: Get File or Folder") faults during +connection resolution because the **executing robot lacks `Connections.View` +permission in the folder where the connection lives** (a different folder than the +one the job runs in). The IS runtime raises `GeneralException` with code +**DAP-GE-3000**. The agent must identify the cross-folder / RBAC cause and +recommend granting `Connections.View` (or moving the connection) — and must NOT +confuse it with a *disabled* connection (`DAP-GE-3005`), an invalid/deleted +connection, an auth failure, or an operation error. + +> **Live-anchored:** built from a **real Orchestrator faulted job** on alpha — a +> Google Drive `ConnectorActivity` published to the Shared folder and run by a +> robot, with the connection bound to a different folder the robot can't read. +> The error text, `Connections.View` guidance, and stack frames are verbatim +> (identities scrubbed). Only the job key/folder envelope is reused as-is. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, real DAP-GE-3000 Connections.View Info) | +| `or jobs logs --level Error` | `or-jobs-logs.json` | +| `or jobs traces ` / `traces spans get --job-key ` | `traces-empty.json` (both forms; real run had no spans) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence +(the Info names the connection's folder and the missing permission). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/39b7d293e1.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/39b7d293e1.json deleted file mode 100644 index 1dae0e02f3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/39b7d293e1.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/9afc8c865b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/9afc8c865b.json deleted file mode 100644 index 4e430ec78a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/9afc8c865b.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/README.md deleted file mode 100644 index cde4a0ea43..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/README.md +++ /dev/null @@ -1,40 +0,0 @@ -# Connector Activity — GeneralException DAP-GE-3000 (robot lacks Connections.View) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the -high-volume `GeneralException DAP-GE-3000` case of TROUB-139 / TROUB-141 -(`ConnectorActivity` / `ConnectorHttpActivity`). - -## What this exercises - -A `ConnectorActivity` ("Google Drive: Get File or Folder") faults during -connection resolution because the **executing robot lacks `Connections.View` -permission in the folder where the connection lives** (a different folder than the -one the job runs in). The IS runtime raises `GeneralException` with code -**DAP-GE-3000**. The agent must identify the cross-folder / RBAC cause and -recommend granting `Connections.View` (or moving the connection) — and must NOT -confuse it with a *disabled* connection (`DAP-GE-3005`), an invalid/deleted -connection, an auth failure, or an operation error. - -> **Live-anchored:** built from a **real Orchestrator faulted job** on alpha — a -> Google Drive `ConnectorActivity` published to the Shared folder and run by a -> robot, with the connection bound to a different folder the robot can't read. -> The error text, `Connections.View` guidance, and stack frames are verbatim -> (identities scrubbed). Only the job key/folder envelope is reused as-is. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, real DAP-GE-3000 Connections.View Info) | -| `or jobs logs --level Error` | `or-jobs-logs.json` | -| `or jobs traces ` / `traces spans get --job-key ` | `traces-empty.json` (both forms; real run had no spans) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence -(the Info names the connection's folder and the missing permission). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/b14c692113.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/b14c692113.json deleted file mode 100644 index bd2935efeb..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/b14c692113.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "f5e5eaa8-ce88-482f-9506-01394caa3967", - "State": "Faulted", - "StartTime": "2026-06-22T14:06:20.220Z", - "EndTime": "2026-06-22T14:06:22.867Z", - "ReleaseName": "DriveSync", - "Source": "Agent", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 4015222 - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/e643049825.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/e643049825.json deleted file mode 100644 index 36573bde4a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/e643049825.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-22T14:06:22.727Z", - "Message": "Google Drive: Get File or Folder: Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: dea5d8d6-ca39-4dbf-b4a5-3516bad38866) Error code: DAP-GE-3000.\r\nin Main.xaml\r\n at ConnectorActivity \"ConnectorActivity\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/edf6406f2b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/edf6406f2b.json deleted file mode 100644 index 98ba513489..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/edf6406f2b.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f5e5eaa8-ce88-482f-9506-01394caa3967", - "StartTime": "2026-06-22T14:06:20.220Z", - "EndTime": "2026-06-22T14:06:22.867Z", - "State": "Faulted", - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "f5e5eaa8-ce88-482f-9506-01394caa3967", - "Info": "Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: dea5d8d6-ca39-4dbf-b4a5-3516bad38866) Error code: DAP-GE-3000.\r\n\r\nin DriveSync.xaml\r\n at ConnectorActivity \"Google Drive: Get File or Folder\"\r\n at Sequence \"DriveSync\"\r\n \r\n\r\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: dea5d8d6-ca39-4dbf-b4a5-3516bad38866) Error code: DAP-GE-3000.\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-22T14:06:19.183Z", - "ReleaseName": "DriveSync", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "ReleaseVersionId": 53490, - "EntryPointPath": "DriveSync.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "robot_account", - "LastModificationTime": "2026-06-22T14:06:22.867Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f5e5eaa8ce88482f950601394caa3967", - "Id": 4015222 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/manifest.json deleted file mode 100644 index 2a55177eda..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/m/r/manifest.json +++ /dev/null @@ -1,55 +0,0 @@ -{ - "version": 2, - "_doc": "LIVE-ANCHORED: built from a real Orchestrator faulted job (a Google Drive ConnectorActivity run in a folder where the robot lacks Connections.View on the connection's folder). GeneralException DAP-GE-3000. Error text is verbatim from the real job, scrubbed. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "39b7d293e1.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "b14c692113.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "b14c692113.json" - }, - { - "match": "or jobs get f5e5eaa8-ce88-482f-9506-01394caa3967", - "file": "edf6406f2b.json" - }, - { - "match": "or jobs logs f5e5eaa8-ce88-482f-9506-01394caa3967", - "file": "e643049825.json" - }, - { - "match": "or jobs traces f5e5eaa8-ce88-482f-9506-01394caa3967", - "file": "9afc8c865b.json" - }, - { - "match": "traces spans get --job-key f5e5eaa8-ce88-482f-9506-01394caa3967", - "file": "9afc8c865b.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get f5e5eaa8-ce88-482f-9506-01394caa3967", - "min": 1, - "description": "Agent fetches the faulted job to read the GeneralException DAP-GE-3000 Connections.View signature and the ConnectorActivity stack frame." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/uip-fixture.json new file mode 100644 index 0000000000..f9c4607b08 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/data/uip-fixture.json @@ -0,0 +1,102 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"f5e5eaa8-ce88-482f-9506-01394caa3967\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T14:06:20.220Z\",\r\n \"EndTime\": \"2026-06-22T14:06:22.867Z\",\r\n \"ReleaseName\": \"DriveSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 4015222\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"f5e5eaa8-ce88-482f-9506-01394caa3967\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T14:06:20.220Z\",\r\n \"EndTime\": \"2026-06-22T14:06:22.867Z\",\r\n \"ReleaseName\": \"DriveSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 4015222\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "f5e5eaa8-ce88-482f-9506-01394caa3967" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f5e5eaa8-ce88-482f-9506-01394caa3967" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f5e5eaa8-ce88-482f-9506-01394caa3967\",\r\n \"StartTime\": \"2026-06-22T14:06:20.220Z\",\r\n \"EndTime\": \"2026-06-22T14:06:22.867Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"f5e5eaa8-ce88-482f-9506-01394caa3967\",\r\n \"Info\": \"Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: dea5d8d6-ca39-4dbf-b4a5-3516bad38866) Error code: DAP-GE-3000.\\r\\n\\r\\nin DriveSync.xaml\\r\\n at ConnectorActivity \\\"Google Drive: Get File or Folder\\\"\\r\\n at Sequence \\\"DriveSync\\\"\\r\\n \\r\\n\\r\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: dea5d8d6-ca39-4dbf-b4a5-3516bad38866) Error code: DAP-GE-3000.\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-22T14:06:19.183Z\",\r\n \"ReleaseName\": \"DriveSync\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"ReleaseVersionId\": 53490,\r\n \"EntryPointPath\": \"DriveSync.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"robot_account\",\r\n \"LastModificationTime\": \"2026-06-22T14:06:22.867Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f5e5eaa8ce88482f950601394caa3967\",\r\n \"Id\": 4015222\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "f5e5eaa8-ce88-482f-9506-01394caa3967" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T14:06:22.727Z\",\r\n \"Message\": \"Google Drive: Get File or Folder: Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: dea5d8d6-ca39-4dbf-b4a5-3516bad38866) Error code: DAP-GE-3000.\\r\\nin Main.xaml\\r\\n at ConnectorActivity \\\"ConnectorActivity\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "f5e5eaa8-ce88-482f-9506-01394caa3967" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get f5e5eaa8-ce88-482f-9506-01394caa3967", + "min": 1, + "description": "Agent fetches the faulted job to read the GeneralException DAP-GE-3000 Connections.View signature and the ConnectorActivity stack frame." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/task.yaml index c2945dfbdb..cfc6ddd00a 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-general-no-access/task.yaml @@ -17,21 +17,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/README.md new file mode 100644 index 0000000000..2333083876 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/README.md @@ -0,0 +1,37 @@ +# Connector Activity — NullReferenceException (null connector output enumerated) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the +`System.NullReferenceException` case of TROUB-139 (`ConnectorActivity`). + +## What this exercises + +A `For Each` iterates the output of an Integration Service connector query +operation that returned **null**, throwing `System.NullReferenceException`. The +exception is opaque (no code, no message detail) — the agent must use the stack +frame (`ForEach\`1` specialized on `IntegrationService.Activities.SWEntities.*_executeQuery_List`) +to conclude it is enumerating a null connector result, and recommend a null/empty +guard. The agent must NOT misattribute it to a connection/auth/operation error. + +Signature pattern (the `ForEach` over an `SWEntities` connector-output type) was +mined from the failed-job telemetry CSV. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, NRE at ForEach) | +| `or jobs logs --level Error` | `or-jobs-logs.json` (names the SWEntities ForEach type) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence +(stack frame identifies the connector-output enumeration). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/31fbeefec3.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/31fbeefec3.json deleted file mode 100644 index 77cb9f1730..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/31fbeefec3.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "State": "Faulted", - "StartTime": "2026-06-19T18:27:40.512Z", - "EndTime": "2026-06-19T18:27:42.137Z", - "ReleaseName": "ContactImport", - "Source": "Agent", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 5039518 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/64792a787a.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/64792a787a.json deleted file mode 100644 index 46c86e0084..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/64792a787a.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "StartTime": "2026-06-19T18:27:40.512Z", - "EndTime": "2026-06-19T18:27:42.137Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "Info": "Object reference not set to an instance of an object.\r\n\r\nin ContactImport.xaml\r\n at ForEach \"For Each contact\"\r\n at Sequence \"ContactImport\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at UiPath.Core.Activities.ForEachBase`1.ContinueExecuting(NativeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-19T18:27:40.512Z", - "ReleaseName": "ContactImport", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "ReleaseVersionId": 14098, - "EntryPointPath": "ContactImport.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-06-19T18:27:42.137Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c2a7f3e18d494b6ca1f27e0d5c3b9a48", - "Id": 5039518 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/6a884494ad.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/6a884494ad.json deleted file mode 100644 index f695163957..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/6a884494ad.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-19T18:27:41.903Z", - "Message": "For Each contact: Object reference not set to an instance of an object.\r\nin ContactImport.xaml\r\n at ForEach \"For Each contact\"\r\n at Sequence \"ContactImport\"\r\n\r\n--- .NET stack trace ---\r\nSystem.NullReferenceException\r\n at UiPath.Core.Activities.ForEachBase`1.ContinueExecuting(NativeActivityContext context)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\r\n--- faulted activity: UiPath.Core.Activities.ForEach`1[[UiPath.IntegrationService.Activities.SWEntities.C5D2F5448FB_executeQuery_List_2M9G3QJHWZK66VJ7LPVRC5298DLOP51.Bundle.executeQuery_List, C5D2F5448FB_executeQ.Tm4Yc2tDOTF1jVNpA3ZT8gl2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]] ---" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/README.md deleted file mode 100644 index 11ba2adf24..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/README.md +++ /dev/null @@ -1,33 +0,0 @@ -# Connector Activity — NullReferenceException (null connector output enumerated) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the -`System.NullReferenceException` case of TROUB-139 (`ConnectorActivity`). - -## What this exercises - -A `For Each` iterates the output of an Integration Service connector query -operation that returned **null**, throwing `System.NullReferenceException`. The -exception is opaque (no code, no message detail) — the agent must use the stack -frame (`ForEach\`1` specialized on `IntegrationService.Activities.SWEntities.*_executeQuery_List`) -to conclude it is enumerating a null connector result, and recommend a null/empty -guard. The agent must NOT misattribute it to a connection/auth/operation error. - -Signature pattern (the `ForEach` over an `SWEntities` connector-output type) was -mined from the failed-job telemetry CSV. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, NRE at ForEach) | -| `or jobs logs --level Error` | `or-jobs-logs.json` (names the SWEntities ForEach type) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence -(stack frame identifies the connector-output enumeration). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/manifest.json deleted file mode 100644 index 8368e20911..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: a ForEach over an Integration Service connector query output (SWEntities.*_executeQuery_List) throws System.NullReferenceException because the connector operation returned null. Signature pattern mined from failed-job telemetry. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "31fbeefec3.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "31fbeefec3.json" - }, - { - "match": "or jobs get c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "file": "64792a787a.json" - }, - { - "match": "or jobs logs c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "file": "6a884494ad.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "min": 1, - "description": "Agent fetches the faulted job to read the NullReferenceException and the ForEach-over-connector-output stack frame." - }, - { - "pattern": "or jobs logs c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", - "min": 0, - "description": "OPTIONAL — error-level logs expose the SWEntities connector-output type being enumerated. min:0 means tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/uip-fixture.json new file mode 100644 index 0000000000..5af19d222c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/data/uip-fixture.json @@ -0,0 +1,84 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-19T18:27:40.512Z\",\r\n \"EndTime\": \"2026-06-19T18:27:42.137Z\",\r\n \"ReleaseName\": \"ContactImport\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5039518\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-19T18:27:40.512Z\",\r\n \"EndTime\": \"2026-06-19T18:27:42.137Z\",\r\n \"ReleaseName\": \"ContactImport\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5039518\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48\",\r\n \"StartTime\": \"2026-06-19T18:27:40.512Z\",\r\n \"EndTime\": \"2026-06-19T18:27:42.137Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48\",\r\n \"Info\": \"Object reference not set to an instance of an object.\\r\\n\\r\\nin ContactImport.xaml\\r\\n at ForEach \\\"For Each contact\\\"\\r\\n at Sequence \\\"ContactImport\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at UiPath.Core.Activities.ForEachBase`1.ContinueExecuting(NativeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-19T18:27:40.512Z\",\r\n \"ReleaseName\": \"ContactImport\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"ReleaseVersionId\": 14098,\r\n \"EntryPointPath\": \"ContactImport.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-06-19T18:27:42.137Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c2a7f3e18d494b6ca1f27e0d5c3b9a48\",\r\n \"Id\": 5039518\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T18:27:41.903Z\",\r\n \"Message\": \"For Each contact: Object reference not set to an instance of an object.\\r\\nin ContactImport.xaml\\r\\n at ForEach \\\"For Each contact\\\"\\r\\n at Sequence \\\"ContactImport\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at UiPath.Core.Activities.ForEachBase`1.ContinueExecuting(NativeActivityContext context)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\\r\\n--- faulted activity: UiPath.Core.Activities.ForEach`1[[UiPath.IntegrationService.Activities.SWEntities.C5D2F5448FB_executeQuery_List_2M9G3QJHWZK66VJ7LPVRC5298DLOP51.Bundle.executeQuery_List, C5D2F5448FB_executeQ.Tm4Yc2tDOTF1jVNpA3ZT8gl2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]] ---\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", + "min": 1, + "description": "Agent fetches the faulted job to read the NullReferenceException and the ForEach-over-connector-output stack frame." + }, + { + "pattern": "or jobs logs c2a7f3e1-8d49-4b6c-a1f2-7e0d5c3b9a48", + "min": 0, + "description": "OPTIONAL — error-level logs expose the SWEntities connector-output type being enumerated. min:0 means tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/task.yaml index 736182747a..896714a3f0 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-null-reference/task.yaml @@ -17,21 +17,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/README.md new file mode 100644 index 0000000000..80d4b04aaa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/README.md @@ -0,0 +1,40 @@ +# Connector Activity — RemoteException token failure (invalid_grant) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers TROUB-140 +(`ConnectorTriggerActivity`) and the `UiPath.Ipc.RemoteException` case. + +## What this exercises + +A `ConnectorTriggerActivity` ("Workday: New Invoice (sample)") faults with +`UiPath.Ipc.RemoteException` wrapping `Could not obtain access token. +(invalid_grant)`. `RemoteException` is the generic out-of-process IPC wrapper, so +the agent must **unwrap to the inner message** to find the real cause: the +connection's OAuth token can't refresh (`invalid_grant`). The fix is to +re-authenticate the connection — NOT to treat it as a generic IPC/network crash. + +This is the disambiguation test: the agent must not stop at "RemoteException" and +must not misclassify it as a transport/platform failure or a disabled connection. + +Signatures (and the `ConnectorTriggerActivity` stack frames) were mined verbatim +from the failed-job telemetry CSV. + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, Info = Ipc.RemoteException / invalid_grant) | +| `or jobs logs --level Error` | `or-jobs-logs.json` | +| `is connections ping ` | `is-connections-ping.json` (AuthenticationFailed, exit 1) | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence +(the unwrapped inner message names the cause). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/20f56540e2.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/20f56540e2.json deleted file mode 100644 index b5cf79c007..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/20f56540e2.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Failure", - "Code": "ConnectionPing", - "Message": "Could not obtain access token. (invalid_grant)", - "Data": { - "Id": "a7c3e1f9-4b6d-42a8-9c05-3e8f1d0b7a62", - "Name": "Workday - Finance", - "ConnectorKey": "uipath-workday", - "IsActive": false, - "State": "AuthenticationFailed" - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/5078159e60.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/5078159e60.json deleted file mode 100644 index 44d4e3cada..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/5078159e60.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", - "State": "Faulted", - "StartTime": "2026-06-20T06:41:33.210Z", - "EndTime": "2026-06-20T06:41:36.882Z", - "ReleaseName": "InvoiceTrigger", - "Source": "Agent", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 5048771 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/736a0767d3.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/736a0767d3.json deleted file mode 100644 index c9733811a5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/736a0767d3.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-20T06:41:36.640Z", - "Message": "Workday: New Invoice (sample): Could not obtain access token. (invalid_grant)\r\nin InvoiceTrigger.xaml\r\n at ConnectorTriggerActivity \"Workday: New Invoice (sample)\"\r\n at Sequence \"InvoiceTrigger\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.Ipc.RemoteException: Could not obtain access token. (invalid_grant)\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.SendAsync(String requestUri, ExecutionParameters executionParameters, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.DebugExecuteAsync(IExecutionService executionService, ExecutionParameters executionParameters, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/README.md deleted file mode 100644 index 522430dbb3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Connector Activity — RemoteException token failure (invalid_grant) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers TROUB-140 -(`ConnectorTriggerActivity`) and the `UiPath.Ipc.RemoteException` case. - -## What this exercises - -A `ConnectorTriggerActivity` ("Workday: New Invoice (sample)") faults with -`UiPath.Ipc.RemoteException` wrapping `Could not obtain access token. -(invalid_grant)`. `RemoteException` is the generic out-of-process IPC wrapper, so -the agent must **unwrap to the inner message** to find the real cause: the -connection's OAuth token can't refresh (`invalid_grant`). The fix is to -re-authenticate the connection — NOT to treat it as a generic IPC/network crash. - -This is the disambiguation test: the agent must not stop at "RemoteException" and -must not misclassify it as a transport/platform failure or a disabled connection. - -Signatures (and the `ConnectorTriggerActivity` stack frames) were mined verbatim -from the failed-job telemetry CSV. - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, Info = Ipc.RemoteException / invalid_grant) | -| `or jobs logs --level Error` | `or-jobs-logs.json` | -| `is connections ping ` | `is-connections-ping.json` (AuthenticationFailed, exit 1) | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence -(the unwrapped inner message names the cause). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/cf0005a862.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/cf0005a862.json deleted file mode 100644 index 630e546295..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/cf0005a862.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", - "StartTime": "2026-06-20T06:41:33.210Z", - "EndTime": "2026-06-20T06:41:36.882Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", - "Info": "Could not obtain access token. (invalid_grant)\r\n\r\nin InvoiceTrigger.xaml\r\n at ConnectorTriggerActivity \"Workday: New Invoice (sample)\"\r\n at Sequence \"InvoiceTrigger\"\r\n\r\nUiPath.Ipc.RemoteException: Could not obtain access token. (invalid_grant)\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.SendAsync(String requestUri, ExecutionParameters executionParameters, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.DebugExecuteAsync(IExecutionService executionService, ExecutionParameters executionParameters, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-20T06:41:33.210Z", - "ReleaseName": "InvoiceTrigger", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "ReleaseVersionId": 7715, - "EntryPointPath": "InvoiceTrigger.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-06-20T06:41:36.882Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "9e1f3a5c7b2d4f6e8a0c1d3b5f7e9a2c", - "Id": 5048771 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/manifest.json deleted file mode 100644 index 572270e645..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/m/r/manifest.json +++ /dev/null @@ -1,57 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: ConnectorTriggerActivity faults with UiPath.Ipc.RemoteException wrapping 'Could not obtain access token. (invalid_grant)'. The operative signal is the unwrapped inner message (OAuth token cannot refresh). Signature mined verbatim from telemetry. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "5078159e60.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "5078159e60.json" - }, - { - "match": "or jobs get 9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", - "file": "cf0005a862.json" - }, - { - "match": "or jobs logs 9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", - "file": "736a0767d3.json" - }, - { - "match": "is connections ping a7c3e1f9-4b6d-42a8-9c05-3e8f1d0b7a62", - "file": "20f56540e2.json", - "exit_code": 1 - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get 9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", - "min": 1, - "description": "Agent fetches the faulted job to read the RemoteException wrapping 'Could not obtain access token. (invalid_grant)' and the ConnectorTriggerActivity stack frame." - }, - { - "pattern": "is connections ping", - "min": 0, - "description": "OPTIONAL — agent may ping the connection to corroborate the auth failure, but the unwrapped inner message already names the cause. min:0 means tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/uip-fixture.json new file mode 100644 index 0000000000..024a2b840b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/data/uip-fixture.json @@ -0,0 +1,95 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T06:41:33.210Z\",\r\n \"EndTime\": \"2026-06-20T06:41:36.882Z\",\r\n \"ReleaseName\": \"InvoiceTrigger\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5048771\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-20T06:41:33.210Z\",\r\n \"EndTime\": \"2026-06-20T06:41:36.882Z\",\r\n \"ReleaseName\": \"InvoiceTrigger\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 5048771\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c\",\r\n \"StartTime\": \"2026-06-20T06:41:33.210Z\",\r\n \"EndTime\": \"2026-06-20T06:41:36.882Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c\",\r\n \"Info\": \"Could not obtain access token. (invalid_grant)\\r\\n\\r\\nin InvoiceTrigger.xaml\\r\\n at ConnectorTriggerActivity \\\"Workday: New Invoice (sample)\\\"\\r\\n at Sequence \\\"InvoiceTrigger\\\"\\r\\n\\r\\nUiPath.Ipc.RemoteException: Could not obtain access token. (invalid_grant)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.SendAsync(String requestUri, ExecutionParameters executionParameters, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.DebugExecuteAsync(IExecutionService executionService, ExecutionParameters executionParameters, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-20T06:41:33.210Z\",\r\n \"ReleaseName\": \"InvoiceTrigger\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"ReleaseVersionId\": 7715,\r\n \"EntryPointPath\": \"InvoiceTrigger.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-06-20T06:41:36.882Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"9e1f3a5c7b2d4f6e8a0c1d3b5f7e9a2c\",\r\n \"Id\": 5048771\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-20T06:41:36.640Z\",\r\n \"Message\": \"Workday: New Invoice (sample): Could not obtain access token. (invalid_grant)\\r\\nin InvoiceTrigger.xaml\\r\\n at ConnectorTriggerActivity \\\"Workday: New Invoice (sample)\\\"\\r\\n at Sequence \\\"InvoiceTrigger\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.Ipc.RemoteException: Could not obtain access token. (invalid_grant)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.SendAsync(String requestUri, ExecutionParameters executionParameters, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.DebugExecuteAsync(IExecutionService executionService, ExecutionParameters executionParameters, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorTriggerActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "a7c3e1f9-4b6d-42a8-9c05-3e8f1d0b7a62" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ConnectionPing\",\r\n \"Message\": \"Could not obtain access token. (invalid_grant)\",\r\n \"Data\": {\r\n \"Id\": \"a7c3e1f9-4b6d-42a8-9c05-3e8f1d0b7a62\",\r\n \"Name\": \"Workday - Finance\",\r\n \"ConnectorKey\": \"uipath-workday\",\r\n \"IsActive\": false,\r\n \"State\": \"AuthenticationFailed\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get 9e1f3a5c-7b2d-4f6e-8a0c-1d3b5f7e9a2c", + "min": 1, + "description": "Agent fetches the faulted job to read the RemoteException wrapping 'Could not obtain access token. (invalid_grant)' and the ConnectorTriggerActivity stack frame." + }, + { + "pattern": "is connections ping", + "min": 0, + "description": "OPTIONAL — agent may ping the connection to corroborate the auth failure, but the unwrapped inner message already names the cause. min:0 means tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/task.yaml index 747492d13a..7f04c68584 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-remote-token/task.yaml @@ -19,21 +19,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/README.md new file mode 100644 index 0000000000..3d5c5aa272 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/README.md @@ -0,0 +1,40 @@ +# Connector Activity — RuntimeException DAP-RT-1101 (operation NotFound) + +Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the +`RuntimeException` case of TROUB-139 (`ConnectorActivity`). + +## What this exercises + +A `ConnectorActivity` ("Google Drive: Get File or Folder") resolves its connection +fine, sends the request, and Google responds **HTTP 404 (File not found)**. The IS +runtime surfaces this as `RuntimeException` with error code **DAP-RT-1101** / +`Status code: NotFound`, carrying a `ProviderMessage` block with the provider's own +404 / `reason=notFound` / `location=fileId`. The agent must distinguish this +**operation-level** failure (missing file / wrong ID) from a connection-resolution +failure (`DAP-GE-*`), and recommend correcting the referenced file identifier. + +> **Live-anchored:** built from a **real Orchestrator faulted job** on alpha — a +> Google Drive `Get File or Folder` ConnectorActivity run by a robot against a live +> connection with a non-existent file ID. The error text (signature, `ProviderMessage`, +> stack frames) and the job envelope are verbatim from that job (identities scrubbed). + +## Mock surface + +| Command | Fixture | +|---|---| +| `or folders list` | `or-folders-list.json` | +| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | +| `or jobs get ` | `or-jobs-get.json` (Faulted, Info = real DAP-RT-1101 NotFound + ProviderMessage) | +| `or jobs logs --level Error` | `or-jobs-logs.json` | +| `docsai ask` | passthrough | + +No project source is staged — the conclusion is reachable from the job evidence +(the failing file ID is in `InputArguments` and the `ProviderMessage`). + +## Success criteria + +`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/3433f64d5d.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/3433f64d5d.json deleted file mode 100644 index 73b948d7f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/3433f64d5d.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", - "State": "Faulted", - "StartTime": "2026-06-22T15:10:10.543Z", - "EndTime": "2026-06-22T15:10:14.957Z", - "ReleaseName": "DriveSync", - "Source": "Agent", - "HostMachineName": "MOCK-HOST", - "ErrorCode": "Robot", - "Id": 4015234 - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/670ba34473.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/670ba34473.json deleted file mode 100644 index 6807a963d4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/670ba34473.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", - "StartTime": "2026-06-22T15:10:10.543Z", - "EndTime": "2026-06-22T15:10:14.957Z", - "State": "Faulted", - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", - "Info": "Request failed with error: \r\nProviderMessage : requestId - 6a395056e4b04e51168424db, providerErrorCode - 404, message - Not Found, providerMessage - error - {code=404, message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., errors=[{message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., domain=global, reason=notFound, location=fileId, locationType=parameter}]}\r\nProviderErrorCode : 404\r\nRequestId : 6a395055e4b03190f1874b62\r\nMessage : Not Found\r\n\r\nStatus code: NotFound. Error code: DAP-RT-1101.\r\n\r\nin DriveSync.xaml\r\n at ConnectorActivity \"Google Drive: Get File or Folder\"\r\n at Sequence \"DriveSync\"\r\n\r\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Request failed with error: \r\nProviderMessage : requestId - 6a395056e4b04e51168424db, providerErrorCode - 404, message - Not Found, providerMessage - error - {code=404, message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., errors=[{message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., domain=global, reason=notFound, location=fileId, locationType=parameter}]}\r\nProviderErrorCode : 404\r\nRequestId : 6a395055e4b03190f1874b62\r\nMessage : Not Found\r\n\r\nStatus code: NotFound. Error code: DAP-RT-1101.\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.GetListResponseAsync(StringBuilder basePath, ExecutionParameters executionParameters, HttpMethod httpMethod, ICollection`1 jitInputs, CancellationToken token, Dictionary`2 telemetryUsedProperties)\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-22T15:10:09.917Z", - "ReleaseName": "DriveSync", - "Type": "Unattended", - "InputArguments": "{\"FileId\":\"1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND\"}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "ReleaseVersionId": 53493, - "EntryPointPath": "DriveSync.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "robot_account", - "LastModificationTime": "2026-06-22T15:10:14.957Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c5d9bf458a554ea3abb2b8a8cca7da02", - "Id": 4015234 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/README.md deleted file mode 100644 index e33e5dc96b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Connector Activity — RuntimeException DAP-RT-1101 (operation NotFound) - -Faithful-replay scenario for the `uipath-troubleshoot` skill. Covers the -`RuntimeException` case of TROUB-139 (`ConnectorActivity`). - -## What this exercises - -A `ConnectorActivity` ("Google Drive: Get File or Folder") resolves its connection -fine, sends the request, and Google responds **HTTP 404 (File not found)**. The IS -runtime surfaces this as `RuntimeException` with error code **DAP-RT-1101** / -`Status code: NotFound`, carrying a `ProviderMessage` block with the provider's own -404 / `reason=notFound` / `location=fileId`. The agent must distinguish this -**operation-level** failure (missing file / wrong ID) from a connection-resolution -failure (`DAP-GE-*`), and recommend correcting the referenced file identifier. - -> **Live-anchored:** built from a **real Orchestrator faulted job** on alpha — a -> Google Drive `Get File or Folder` ConnectorActivity run by a robot against a live -> connection with a non-existent file ID. The error text (signature, `ProviderMessage`, -> stack frames) and the job envelope are verbatim from that job (identities scrubbed). - -## Mock surface - -| Command | Fixture | -|---|---| -| `or folders list` | `or-folders-list.json` | -| `or jobs list --folder-key [--state Faulted]` | `or-jobs-list-faulted.json` | -| `or jobs get ` | `or-jobs-get.json` (Faulted, Info = real DAP-RT-1101 NotFound + ProviderMessage) | -| `or jobs logs --level Error` | `or-jobs-logs.json` | -| `docsai ask` | passthrough | - -No project source is staged — the conclusion is reachable from the job evidence -(the failing file ID is in `InputArguments` and the `ProviderMessage`). - -## Success criteria - -`skill_triggered` + `llm_judge` (graded against `RESOLUTION.md`, final response only). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/f8f2069c90.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/f8f2069c90.json deleted file mode 100644 index 3890d195d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/f8f2069c90.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-22T15:10:14.838Z", - "Message": "Google Drive: Get File or Folder: Request failed with error: \r\nProviderMessage : requestId - 6a395056e4b04e51168424db, providerErrorCode - 404, message - Not Found, providerMessage - error - {code=404, message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., errors=[{message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., domain=global, reason=notFound, location=fileId, locationType=parameter}]}\r\nProviderErrorCode : 404\r\nRequestId : 6a395055e4b03190f1874b62\r\nMessage : Not Found\r\n\r\nStatus code: NotFound. Error code: DAP-RT-1101.\r\nin DriveSync.xaml\r\n at ConnectorActivity \"Google Drive: Get File or Folder\"\r\n at Sequence \"DriveSync\"\r\n\r\n--- .NET stack trace ---\r\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.GetListResponseAsync(StringBuilder basePath, ExecutionParameters executionParameters, HttpMethod httpMethod, ICollection`1 jitInputs, CancellationToken token, Dictionary`2 telemetryUsedProperties)\r\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\r\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/manifest.json deleted file mode 100644 index 96354e7428..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/m/r/manifest.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful replay: ConnectorActivity faults with RuntimeException DAP-RT-1101 (connector operation returned NotFound). Signature mined verbatim from failed-job telemetry. First-match-wins.", - "rules": [ - { - "match": "or folders list", - "file": "897a8be769.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --state Faulted", - "file": "3433f64d5d.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "file": "3433f64d5d.json" - }, - { - "match": "or jobs get c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", - "file": "670ba34473.json" - }, - { - "match": "or jobs logs c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", - "file": "f8f2069c90.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "min": 1, - "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." - }, - { - "pattern": "or jobs get c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", - "min": 1, - "description": "Agent fetches the faulted job to read the RuntimeException DAP-RT-1101 'Status code: NotFound' signature and the ConnectorActivity stack frame." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/uip-fixture.json new file mode 100644 index 0000000000..3c19e7708c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/data/uip-fixture.json @@ -0,0 +1,79 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T15:10:10.543Z\",\r\n \"EndTime\": \"2026-06-22T15:10:14.957Z\",\r\n \"ReleaseName\": \"DriveSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 4015234\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02\",\r\n \"State\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T15:10:10.543Z\",\r\n \"EndTime\": \"2026-06-22T15:10:14.957Z\",\r\n \"ReleaseName\": \"DriveSync\",\r\n \"Source\": \"Agent\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ErrorCode\": \"Robot\",\r\n \"Id\": 4015234\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02\",\r\n \"StartTime\": \"2026-06-22T15:10:10.543Z\",\r\n \"EndTime\": \"2026-06-22T15:10:14.957Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02\",\r\n \"Info\": \"Request failed with error: \\r\\nProviderMessage : requestId - 6a395056e4b04e51168424db, providerErrorCode - 404, message - Not Found, providerMessage - error - {code=404, message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., errors=[{message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., domain=global, reason=notFound, location=fileId, locationType=parameter}]}\\r\\nProviderErrorCode : 404\\r\\nRequestId : 6a395055e4b03190f1874b62\\r\\nMessage : Not Found\\r\\n\\r\\nStatus code: NotFound. Error code: DAP-RT-1101.\\r\\n\\r\\nin DriveSync.xaml\\r\\n at ConnectorActivity \\\"Google Drive: Get File or Folder\\\"\\r\\n at Sequence \\\"DriveSync\\\"\\r\\n\\r\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Request failed with error: \\r\\nProviderMessage : requestId - 6a395056e4b04e51168424db, providerErrorCode - 404, message - Not Found, providerMessage - error - {code=404, message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., errors=[{message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., domain=global, reason=notFound, location=fileId, locationType=parameter}]}\\r\\nProviderErrorCode : 404\\r\\nRequestId : 6a395055e4b03190f1874b62\\r\\nMessage : Not Found\\r\\n\\r\\nStatus code: NotFound. Error code: DAP-RT-1101.\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.GetListResponseAsync(StringBuilder basePath, ExecutionParameters executionParameters, HttpMethod httpMethod, ICollection`1 jitInputs, CancellationToken token, Dictionary`2 telemetryUsedProperties)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-22T15:10:09.917Z\",\r\n \"ReleaseName\": \"DriveSync\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{\\\"FileId\\\":\\\"1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND\\\"}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"ReleaseVersionId\": 53493,\r\n \"EntryPointPath\": \"DriveSync.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"robot_account\",\r\n \"LastModificationTime\": \"2026-06-22T15:10:14.957Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c5d9bf458a554ea3abb2b8a8cca7da02\",\r\n \"Id\": 4015234\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T15:10:14.838Z\",\r\n \"Message\": \"Google Drive: Get File or Folder: Request failed with error: \\r\\nProviderMessage : requestId - 6a395056e4b04e51168424db, providerErrorCode - 404, message - Not Found, providerMessage - error - {code=404, message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., errors=[{message=File not found: 1AbCdEfGhIjKlMnOpQrStUvWxYz000NOTFOUND., domain=global, reason=notFound, location=fileId, locationType=parameter}]}\\r\\nProviderErrorCode : 404\\r\\nRequestId : 6a395055e4b03190f1874b62\\r\\nMessage : Not Found\\r\\n\\r\\nStatus code: NotFound. Error code: DAP-RT-1101.\\r\\nin DriveSync.xaml\\r\\n at ConnectorActivity \\\"Google Drive: Get File or Folder\\\"\\r\\n at Sequence \\\"DriveSync\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.GetListResponseAsync(StringBuilder basePath, ExecutionParameters executionParameters, HttpMethod httpMethod, ICollection`1 jitInputs, CancellationToken token, Dictionary`2 telemetryUsedProperties)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\r\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "min": 1, + "description": "Agent lists faulted jobs in the Shared folder to find the failing job (prompt gives no job key)." + }, + { + "pattern": "or jobs get c5d9bf45-8a55-4ea3-abb2-b8a8cca7da02", + "min": 1, + "description": "Agent fetches the faulted job to read the RuntimeException DAP-RT-1101 'Status code: NotFound' signature and the ConnectorActivity stack frame." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/task.yaml index 26261f4ee9..2f50a886f0 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/connector-runtime-notfound/task.yaml @@ -18,21 +18,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/README.md new file mode 100644 index 0000000000..82fb064a3a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/README.md @@ -0,0 +1,31 @@ +# is-activities-prerelease-not-found + +Hand-authored **design-time** scenario for the **IS Activities Prerelease/Beta Not Found** playbook +([`is-activities-prerelease-not-found.md`](../../../../../skills/uipath-troubleshoot/references/products/integration-service/playbooks/is-activities-prerelease-not-found.md)). + +## What the scenario exercises + +A Studio project (`AccountSync`) whose `project.json` pins `UiPath.IntegrationService.Activities` to a +prerelease build (`1.16.0-beta.20250603.1`) the configured feed does not serve. Dependency restore fails and +every Integration Service activity shows invalid. The agent must diagnose the **prerelease-not-in-feed** cause +(pulled by the unified package's auto-upgrade-on-open) and recommend connecting to the official cloud +env / UiPath-Official feed and pinning a **stable** version. + +No Orchestrator job exists — this is a design-time restore failure, so the agent investigates by reading the +project source (`project.json`), not by querying `uip or jobs …`. + +## Why the two criteria stay aligned (no bypass, correct routing) + +- The exact `-beta` version lives in `project.json`, not the prompt — the agent must Read the source to find it. +- The prompt uses **regression + diagnostic** framing ("worked last week … figure out why it broke") to route to + `uipath-troubleshoot` rather than a build/fix skill. +- The correct fix (official feed / stable pin / disable prerelease, and *why* the auto-upgrade re-pinned a beta) + requires the playbook, so a shallow "just pick another version" answer lands below the `llm_judge` threshold. + +## Mock + +(there is no job to query). Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/m/r/README.md deleted file mode 100644 index c674a39746..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/m/r/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# is-activities-prerelease-not-found - -Hand-authored **design-time** scenario for the **IS Activities Prerelease/Beta Not Found** playbook -([`is-activities-prerelease-not-found.md`](../../../../../skills/uipath-troubleshoot/references/products/integration-service/playbooks/is-activities-prerelease-not-found.md)). - -## What the scenario exercises - -A Studio project (`AccountSync`) whose `project.json` pins `UiPath.IntegrationService.Activities` to a -prerelease build (`1.16.0-beta.20250603.1`) the configured feed does not serve. Dependency restore fails and -every Integration Service activity shows invalid. The agent must diagnose the **prerelease-not-in-feed** cause -(pulled by the unified package's auto-upgrade-on-open) and recommend connecting to the official cloud -env / UiPath-Official feed and pinning a **stable** version. - -No Orchestrator job exists — this is a design-time restore failure, so the agent investigates by reading the -project source (`project.json`), not by querying `uip or jobs …`. - -## Why the two criteria stay aligned (no bypass, correct routing) - -- The exact `-beta` version lives in `project.json`, not the prompt — the agent must Read the source to find it. -- The prompt uses **regression + diagnostic** framing ("worked last week … figure out why it broke") to route to - `uipath-troubleshoot` rather than a build/fix skill. -- The correct fix (official feed / stable pin / disable prerelease, and *why* the auto-upgrade re-pinned a beta) - requires the playbook, so a shallow "just pick another version" answer lands below the `llm_judge` threshold. - -## Mock - -Source-driven. `data/m/r/manifest.json` proxies only `docsai ask`; every other `uip` call returns an empty array -(there is no job to query). Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/m/r/manifest.json deleted file mode 100644 index 1fad99f5d1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored design-time (no-job) scenario. Only docsai ask is proxied; any other uip command returns an empty array.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Open-ended natural-language query; proxy to the real uip CLI per troubleshoot test rules." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting. This is a design-time issue — there is no Orchestrator job to query." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/uip-fixture.json new file mode 100644 index 0000000000..bcb5501faa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/data/uip-fixture.json @@ -0,0 +1,8 @@ +{ + "version": 1, + "responses": [], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/task.yaml index de92315b68..1f97dcf7c6 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/is-activities-prerelease-not-found/task.yaml @@ -15,22 +15,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/README.md new file mode 100644 index 0000000000..9128234487 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/README.md @@ -0,0 +1,60 @@ +# rpa-is-connection-disabled-dap — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim `uip` CLI +responses captured from that session. + +## What the original session uncovered + +An Orchestrator unattended job (`TokenRefreshFailedJob`) in the Solution folder +`Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime` faulted with +Integration Service error **`DAP-GE-3005` (ConnectionDisabled)** at the +`ConnectorActivity "Create Folder"` in `Main.xaml`. Despite the folder's name +("TokenRefreshFailed"), the emitted runtime code is `DAP-GE-3005`, **not** +`DAP-GE-3004` — the diagnosis must follow the evidence, not the label. The root +cause was the **Box** connection bound to that activity +(`989fd9d2-...`, in folder `b262689d-...`), which pinged as **`Failed` / +not enabled** — auto-disabled following an OAuth re-authentication failure. The +fix (re-authenticating the connection) was verified: a re-run completed +Successfully. See `RESOLUTION.md` for the full ground truth. + +The test exercises the agent's ability to (a) trust the emitted DAP code over +the folder name, (b) recover the connection ID from the job's +`ResourceOverwrites`, and (c) confirm the disabled/failed connection state and +recommend re-authentication. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +This investigation was purely CLI-driven (the project is a Studio Web project, +no local source), so there is **no** `process/` snapshot. + +> Note: connection-state-dependent fixtures (`is connections ping 989fd9d2-...` +> and the bound-folder `is connections list --folder-key b262689d-...`) capture +> the **pre-fix `Failed` state** at investigation time — not the post-fix +> `Enabled` state the live tenant shows now. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). +- An LLM judge grades the agent's final answer against `RESOLUTION.md`: correct + root cause (disabled/`Failed` Box connection → `DAP-GE-3005`) **and** correct + fix (re-authenticate / re-enable the connection). + +## Re-running the extraction + +If the source transcript changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation <.local/investigations dir> --transcript \ + --resolution RESOLUTION.md --scenario-name rpa-is-connection-disabled-dap --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/02e8acc99f.json deleted file mode 100644 index 6205d04188..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/081b84c7c0.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/081b84c7c0.json deleted file mode 100644 index 3d67262d3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/081b84c7c0.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "Message", - "Data": { - "Message": "No connections found for any connector in folder 'eba26f86-ea3a-4955-a029-28c6e5006dab'." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/23cb6207f5.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/23cb6207f5.json deleted file mode 100644 index adc52aabf3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/23cb6207f5.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-856b-c1dd9e521e15", - "TraceId": "3baa335d-8cd1-44cf-936c-df8c013c99df", - "ParentId": null, - "Name": "Robot Job 3baa335d-8cd1-44cf-936c-df8c013c99df", - "StartTime": "2026-06-19T11:49:58.2096486Z", - "EndTime": "2026-06-19T11:50:00.5096756Z", - "Attributes": "{\"Exception\":\"UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "311c6049-4966-4ef2-a6f3-8ceccaf38d54", - "TenantId": "69d43765-6ede-4575-88b7-9db601a2c056", - "ExpiryTimeUtc": null, - "FolderKey": "eba26f86-ea3a-4955-a029-28c6e5006dab", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "f9ecba5f-8dd8-4505-aeee-76ec5777d954", - "JobKey": "3baa335d-8cd1-44cf-936c-df8c013c99df", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-19T11:50:00.7904098Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/2e37ebc77e.json deleted file mode 100644 index 44bfb742f1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,406 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "TokenRefreshFailed_DAPRuntime", - "ID": 2979245, - "Key": "eba26f86-ea3a-4955-a029-28c6e5006dab", - "Path": "Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-folder-b-b4b03014", - "ID": 2978732, - "Key": "771ec4f1-f031-463e-a6b5-5ebfec25ba86", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-b4b03014", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-deploy-folder-4ab5f95e", - "ID": 2976398, - "Key": "59a72d90-70b6-4272-a665-325b0eaf9d62", - "Path": "Shared/uipath-platform-e2e/e2e-deploy-folder-4ab5f95e", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-e5b01329", - "ID": 2976354, - "Key": "c2bba2f7-5f5f-4bfd-9781-ff7b5e11efef", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-e5b01329", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-8c88e3dc-B", - "ID": 2976154, - "Key": "cacc23b2-b067-4191-9ea1-b73b3b2e98a0", - "Path": "Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-8c88e3dc-A", - "ID": 2976153, - "Key": "10908d6a-f2a8-4296-a034-d91137aee4cb", - "Path": "Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-8c88e3dc-top", - "ID": 2976151, - "Key": "a027aeca-4f5d-4fb3-92f8-b8e97619b595", - "Path": "Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-72c44bd7", - "ID": 2976049, - "Key": "fb61e5ba-40ef-4aca-9830-0a86dd6cfdae", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-72c44bd7", - "Type": "Standard", - "Description": "" - }, - { - "Name": "RequestFailed_DAPRuntime", - "ID": 2976014, - "Key": "e555fecd-20d3-47aa-864c-1c2e588d4769", - "Path": "Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime", - "Type": "Solution", - "Description": "" - }, - { - "Name": "uipath-rpa-isActivities", - "ID": 2975935, - "Key": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Path": "Shared/uipath-rpa-isActivities", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-B", - "ID": 2975612, - "Key": "94f779c1-f602-49d1-a253-8a6bc18c0c45", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-A", - "ID": 2975609, - "Key": "fb57fea0-6855-4bd2-8b86-3bc9332a545f", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-top", - "ID": 2975605, - "Key": "7f121d04-2efb-4a6e-964a-2b00c7ac41e5", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-5b56c68d", - "ID": 2975488, - "Key": "3cbad2ee-ba8a-4d87-8048-985d7bb95336", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-5b56c68d", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-00fb23da", - "ID": 2975403, - "Key": "95d78e99-e937-417f-8e0c-82d6e46c04bd", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-00fb23da", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-22d1b6dd", - "ID": 2974296, - "Key": "2292cfbc-81ef-4252-a991-554de4bf2d48", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-22d1b6dd", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-acdd13ce", - "ID": 2974264, - "Key": "e1e0b094-425b-4a36-a7ec-0fe528e6ac1c", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-acdd13ce", - "Type": "Standard", - "Description": "" - }, - { - "Name": "CaseTest", - "ID": 2973309, - "Key": "583f75bc-3630-4d55-b269-5c8af1f17b80", - "Path": "Shared/uipath-maestro-case/CaseTest", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-deploy-folder-7188808a", - "ID": 2973247, - "Key": "40d377a0-0938-4e07-ad53-3e4b1f4dccdb", - "Path": "Shared/uipath-platform-e2e/e2e-deploy-folder-7188808a", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-folder-b-d6910c88", - "ID": 2973163, - "Key": "c1ddde01-465d-4684-a603-b2b96d87af94", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-d6910c88", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-0875838e", - "ID": 2973103, - "Key": "9cabacb6-8568-46e8-8cca-21a46f1a236b", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-0875838e", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-B", - "ID": 2973045, - "Key": "a909fdc9-4d95-48ab-9cf3-1f1c7e60d5de", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-A", - "ID": 2973044, - "Key": "c14bac5f-e6f0-4cc1-bb72-486f229421d2", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-top", - "ID": 2973043, - "Key": "37fc9a90-9d49-414a-9158-c883eb0cd43b", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-B", - "ID": 2955039, - "Key": "e66dbaa5-d7cf-4b9e-b66c-d42b372a2df1", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-A", - "ID": 2955037, - "Key": "46cc134a-e616-47ec-9e9a-360bf905ed5d", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-top", - "ID": 2955036, - "Key": "d6cd4cc4-a00e-4b87-8c46-4044385ddb3f", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-8df5d996", - "ID": 2954999, - "Key": "cd615438-aad8-4efe-af56-557bad7c1c2c", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-8df5d996", - "Type": "Standard", - "Description": "E2E test folder B for asset sharing" - }, - { - "Name": "e2e-folder-b-fce8a341", - "ID": 2954890, - "Key": "ad9bf36f-32e3-42b8-b051-c6cb114902ea", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-fce8a341", - "Type": "Standard", - "Description": "E2E test folder B for asset scoping isolation check" - }, - { - "Name": "e2e-fh-0b53372d-B", - "ID": 2952698, - "Key": "a1ef65e1-0213-4b50-ac52-e00b8235cad9", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-0b53372d-A", - "ID": 2952696, - "Key": "7a03267f-ec72-4cf6-a7a0-aa9128d8389a", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-0b53372d-top", - "ID": 2952694, - "Key": "c6f01161-4e94-4c59-a85a-76ac9b14963d", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-372553a0", - "ID": 2952623, - "Key": "5acebb10-f8d0-4c9a-a732-651f0f77a6ac", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-372553a0", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-e7972fc2", - "ID": 2952381, - "Key": "0347966b-d3db-4da4-b4ce-055d1b9a70a7", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-e7972fc2", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Solution 7", - "ID": 2933058, - "Key": "2c3f72f3-aaad-4cf6-b1bc-9e6504a3cb04", - "Path": "Shared/Solution 7", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-dr-debug", - "ID": 2923711, - "Key": "993f1502-37ff-43f4-81d9-3c6b040d8bbf", - "Path": "Shared/uipath-maestro-flow/Example-dr-debug", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-dailyreport", - "ID": 2923530, - "Key": "1c8e5662-0616-410f-b9e7-c01003b467c2", - "Path": "Shared/uipath-maestro-flow/Example-dailyreport", - "Type": "Solution", - "Description": "" - }, - { - "Name": "ExampleV3", - "ID": 2923525, - "Key": "1fc080e4-a221-44fe-a9b5-9f526637502f", - "Path": "Shared/ExampleV3", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-debug", - "ID": 2923476, - "Key": "26315131-b943-4d72-9442-3476a0196bec", - "Path": "Shared/uipath-maestro-flow/Example-debug", - "Type": "Solution", - "Description": "" - }, - { - "Name": "FraudEscalation", - "ID": 2922399, - "Key": "8492394e-2f77-4b7d-96c1-c0b0662e3819", - "Path": "Shared/uipath-agents/FraudEscalation", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-B", - "ID": 2901931, - "Key": "fefa7912-d5d3-4dd5-87ea-3c4e3813321f", - "Path": "Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-A", - "ID": 2901928, - "Key": "44fd0e78-9eda-4a45-84f5-b2e07473f312", - "Path": "Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-top", - "ID": 2901927, - "Key": "ffd6cb46-e6b8-4c58-94b4-3ebd480903c0", - "Path": "Shared/e2e-fh-eecfa769-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Finance", - "ID": 2901921, - "Key": "fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed", - "Path": "Finance", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-a-0a0a527c", - "ID": 2901912, - "Key": "c8c3f842-ef1f-4e9a-9728-191cba4bfd89", - "Path": "Shared/e2e-folder-a-0a0a527c", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-0a0a527c", - "ID": 2901910, - "Key": "0098b403-912d-48bf-983b-0bf7cc7a1240", - "Path": "Shared/e2e-folder-b-0a0a527c", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-a-8805f811", - "ID": 2901871, - "Key": "f573d08c-c8fa-485b-ad39-917b502eca45", - "Path": "Shared/e2e-folder-a-8805f811", - "Type": "Standard", - "Description": "E2E test folder A for asset sharing" - }, - { - "Name": "e2e-folder-b-8805f811", - "ID": 2901870, - "Key": "66247ea0-e4ad-4eb5-989d-846926c0acd5", - "Path": "Shared/e2e-folder-b-8805f811", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-a441329c", - "ID": 2901830, - "Key": "b5eb7093-74cf-443c-8d02-8712f4831c5a", - "Path": "Shared/e2e-folder-b-a441329c", - "Type": "Standard", - "Description": "E2E test folder B for asset sharing" - }, - { - "Name": "e2e-folder-a-a441329c", - "ID": 2901829, - "Key": "616562ea-0b92-41bc-9c3e-110c3981a647", - "Path": "Shared/e2e-folder-a-a441329c", - "Type": "Standard", - "Description": "E2E test folder A for asset sharing" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/4dc2cb914e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/4dc2cb914e.json deleted file mode 100644 index 379f57202e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/4dc2cb914e.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "Result": "Failure", - "Code": "ConnectionNotEnabled", - "Message": "Connection '989fd9d2-4b07-443e-86e3-ed6f1b00079b' is not enabled (status: Failed).", - "Instructions": "Run 'uip is connections edit ' to re-authenticate, or 'uip is connections create ' to create a new connection." -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/5d05e6495f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/5d05e6495f.json deleted file mode 100644 index 444eb501ea..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/5d05e6495f.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-19T11:50:00.760Z", - "JobId": 50250322 - }, - { - "State": "Running", - "Timestamp": "2026-06-19T11:49:55.433Z", - "JobId": 50250322 - }, - { - "State": "Pending", - "Timestamp": "2026-06-19T11:49:52.307Z", - "JobId": 50250322 - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/969f49689d.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/969f49689d.json deleted file mode 100644 index cd6f7fbdf8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/969f49689d.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "3baa335d-8cd1-44cf-936c-df8c013c99df", - "StartTime": "2026-06-19T11:49:55.423Z", - "EndTime": "2026-06-19T11:50:00.733Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "ResourceOverwrites": "{\"Connection.989fd9d2-4b07-443e-86e3-ed6f1b00079b\":{\"EntityDisplayName\":\"original_email@test.com\",\"Level\":0,\"Properties\":{\"connectionId\":\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\",\"folderKey\":\"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\"ConnectionId\":\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\"}}}", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f730fadc-abfb-4c12-a627-915f8a99e224", - "Info": "Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\r\n\r\nin Main.xaml\n at ConnectorActivity \"Create Folder\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-19T11:49:52.263Z", - "ReleaseName": "TokenRefreshFailedJob", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "B6818817-9380-4F00-6A5A-0B49C962BBA9", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Serverless", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1089281, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 2979245, - "FolderKey": "eba26f86-ea3a-4955-a029-28c6e5006dab", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": true, - "LocalSystemAccount": "ROBOTUSER", - "OrchestratorUserIdentity": "replacement_email@test.com", - "RemoteControlAccess": "None", - "ServerlessJobType": "RobotJob", - "LastModificationTime": "2026-06-19T11:50:00.733Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "3baa335d8cd144cf936cdf8c013c99df", - "Id": 50250322 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/README.md deleted file mode 100644 index d9f78390e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/README.md +++ /dev/null @@ -1,59 +0,0 @@ -# rpa-is-connection-disabled-dap — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim `uip` CLI -responses captured from that session. - -## What the original session uncovered - -An Orchestrator unattended job (`TokenRefreshFailedJob`) in the Solution folder -`Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime` faulted with -Integration Service error **`DAP-GE-3005` (ConnectionDisabled)** at the -`ConnectorActivity "Create Folder"` in `Main.xaml`. Despite the folder's name -("TokenRefreshFailed"), the emitted runtime code is `DAP-GE-3005`, **not** -`DAP-GE-3004` — the diagnosis must follow the evidence, not the label. The root -cause was the **Box** connection bound to that activity -(`989fd9d2-...`, in folder `b262689d-...`), which pinged as **`Failed` / -not enabled** — auto-disabled following an OAuth re-authentication failure. The -fix (re-authenticating the connection) was verified: a re-run completed -Successfully. See `RESOLUTION.md` for the full ground truth. - -The test exercises the agent's ability to (a) trust the emitted DAP code over -the folder name, (b) recover the connection ID from the job's -`ResourceOverwrites`, and (c) confirm the disabled/failed connection state and -recommend re-authentication. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real `uip` stdout captured verbatim from the session | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -This investigation was purely CLI-driven (the project is a Studio Web project, -no local source), so there is **no** `process/` snapshot. - -> Note: connection-state-dependent fixtures (`is connections ping 989fd9d2-...` -> and the bound-folder `is connections list --folder-key b262689d-...`) capture -> the **pre-fix `Failed` state** at investigation time — not the post-fix -> `Enabled` state the live tenant shows now. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). -- An LLM judge grades the agent's final answer against `RESOLUTION.md`: correct - root cause (disabled/`Failed` Box connection → `DAP-GE-3005`) **and** correct - fix (re-authenticate / re-enable the connection). - -## Re-running the extraction - -If the source transcript changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation <.local/investigations dir> --transcript \ - --resolution RESOLUTION.md --scenario-name rpa-is-connection-disabled-dap --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/d30940f343.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/d30940f343.json deleted file mode 100644 index eee2a5da03..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/d30940f343.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "2524b63d-0730-4feb-9e5a-fc9a7f7a23f1", - "State": "Successful", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "C1ED99D1-5381-4F00-C7BD-0B49C96798C9", - "StartTime": "2026-06-19T12:21:56.340Z", - "EndTime": "2026-06-19T12:22:02.817Z", - "CreationTime": "2026-06-19T12:21:53.570Z", - "Id": 50251460 - }, - { - "Key": "3baa335d-8cd1-44cf-936c-df8c013c99df", - "State": "Faulted", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "B6818817-9380-4F00-6A5A-0B49C962BBA9", - "StartTime": "2026-06-19T11:49:55.423Z", - "EndTime": "2026-06-19T11:50:00.733Z", - "CreationTime": "2026-06-19T11:49:52.263Z", - "Id": 50250322 - }, - { - "Key": "0a2e8a22-ccb4-4953-9356-151d32a7e851", - "State": "Faulted", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "8A852950-F07E-4F00-AC73-0B49C95A9A71", - "StartTime": "2026-06-19T10:51:37.240Z", - "EndTime": "2026-06-19T10:51:45.237Z", - "CreationTime": "2026-06-19T10:51:32.910Z", - "Id": 50248422 - }, - { - "Key": "2c0c44a4-364c-4cd8-934d-7c73bb3ff354", - "State": "Successful", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "15117404-EA7A-4F00-444C-0B49C94419E8", - "StartTime": "2026-06-19T08:07:56.780Z", - "EndTime": "2026-06-19T08:08:03.683Z", - "CreationTime": "2026-06-19T08:07:45.107Z", - "Id": 50242038 - } - ], - "Pagination": { - "Returned": 4, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/e919f3989b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/e919f3989b.json deleted file mode 100644 index ae247368cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/e919f3989b.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "ValidationError", - "Message": "error: unknown option '--all'", - "Instructions": "Check command arguments and options. Use --help for usage information." -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/f0a4db5418.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/f0a4db5418.json deleted file mode 100644 index b51ac98930..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/f0a4db5418.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-19T11:50:00.487Z", - "Message": "Create Folder: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\nin Main.xaml\n at ConnectorActivity \"Create Folder\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/f50db666cc.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/f50db666cc.json deleted file mode 100644 index a2d087806a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/f50db666cc.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "29aae689-111d-459f-9a4a-d1ad1d9d2761", - "Name": "uipath", - "ConnectorKey": "uipath-freshworks-freshdesk", - "ConnectorName": "Freshdesk", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "No", - "Folder": "uipath-rpa-isActivities", - "FolderKey": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Created": "2026-06-18T13:24:02.950Z", - "Updated": "2026-06-19T11:50:51.870Z" - }, - { - "Id": "989fd9d2-4b07-443e-86e3-ed6f1b00079b", - "Name": "original_email@test.com", - "ConnectorKey": "uipath-box-box", - "ConnectorName": "Box", - "State": "Failed", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "uipath-rpa-isActivities", - "FolderKey": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Created": "2026-06-19T07:46:21.196Z", - "Updated": "2026-06-19T12:08:33.426Z" - } - ], - "Warning": "Connection(s) not enabled: 989fd9d2-4b07-443e-86e3-ed6f1b00079b. Only Enabled connections can be used for operations. Run 'uip is connections edit ' to re-authenticate." -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/manifest.json deleted file mode 100644 index 3e3f2cf01f..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/m/r/manifest.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-replay dispatch table for the rpa-is-connection-disabled-dap scenario. Fixtures are verbatim uip CLI responses captured from the real session. Connection-state-dependent fixtures (ping / bound-folder list) reflect the PRE-FIX 'Failed' state at investigation time.", - "rules": [ - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list --all --output json", - "file": "e919f3989b.json", - "exit_code": 3 - }, - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key eba26f86-ea3a-4955-a029-28c6e5006dab --output json", - "file": "d30940f343.json" - }, - { - "match": "or jobs get 3baa335d-8cd1-44cf-936c-df8c013c99df --output json", - "file": "969f49689d.json" - }, - { - "match": "or jobs logs 3baa335d-8cd1-44cf-936c-df8c013c99df --level Error --output json", - "file": "f0a4db5418.json" - }, - { - "match": "or jobs history 3baa335d-8cd1-44cf-936c-df8c013c99df --output json", - "file": "5d05e6495f.json" - }, - { - "match": "or jobs traces 3baa335d-8cd1-44cf-936c-df8c013c99df --output json", - "file": "23cb6207f5.json" - }, - { - "match": "is connections list --folder-key eba26f86-ea3a-4955-a029-28c6e5006dab --output json", - "file": "081b84c7c0.json" - }, - { - "match": "is connections list --folder-key b262689d-e2a9-4c1a-8e68-04382a7fff76 --output json", - "file": "f50db666cc.json" - }, - { - "match": "is connections ping 989fd9d2-4b07-443e-86e3-ed6f1b00079b --output json", - "file": "4dc2cb914e.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/uip-fixture.json new file mode 100644 index 0000000000..fcedff033b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/data/uip-fixture.json @@ -0,0 +1,151 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "3baa335d-8cd1-44cf-936c-df8c013c99df", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T11:50:00.487Z\",\r\n \"Message\": \"Create Folder: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\nin Main.xaml\\n at ConnectorActivity \\\"Create Folder\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "eba26f86-ea3a-4955-a029-28c6e5006dab", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"2524b63d-0730-4feb-9e5a-fc9a7f7a23f1\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"C1ED99D1-5381-4F00-C7BD-0B49C96798C9\",\r\n \"StartTime\": \"2026-06-19T12:21:56.340Z\",\r\n \"EndTime\": \"2026-06-19T12:22:02.817Z\",\r\n \"CreationTime\": \"2026-06-19T12:21:53.570Z\",\r\n \"Id\": 50251460\r\n },\r\n {\r\n \"Key\": \"3baa335d-8cd1-44cf-936c-df8c013c99df\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"B6818817-9380-4F00-6A5A-0B49C962BBA9\",\r\n \"StartTime\": \"2026-06-19T11:49:55.423Z\",\r\n \"EndTime\": \"2026-06-19T11:50:00.733Z\",\r\n \"CreationTime\": \"2026-06-19T11:49:52.263Z\",\r\n \"Id\": 50250322\r\n },\r\n {\r\n \"Key\": \"0a2e8a22-ccb4-4953-9356-151d32a7e851\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"8A852950-F07E-4F00-AC73-0B49C95A9A71\",\r\n \"StartTime\": \"2026-06-19T10:51:37.240Z\",\r\n \"EndTime\": \"2026-06-19T10:51:45.237Z\",\r\n \"CreationTime\": \"2026-06-19T10:51:32.910Z\",\r\n \"Id\": 50248422\r\n },\r\n {\r\n \"Key\": \"2c0c44a4-364c-4cd8-934d-7c73bb3ff354\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"15117404-EA7A-4F00-444C-0B49C94419E8\",\r\n \"StartTime\": \"2026-06-19T08:07:56.780Z\",\r\n \"EndTime\": \"2026-06-19T08:08:03.683Z\",\r\n \"CreationTime\": \"2026-06-19T08:07:45.107Z\",\r\n \"Id\": 50242038\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 4,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "eba26f86-ea3a-4955-a029-28c6e5006dab", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Message\",\r\n \"Data\": {\r\n \"Message\": \"No connections found for any connector in folder 'eba26f86-ea3a-4955-a029-28c6e5006dab'.\"\r\n }\r\n}" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "b262689d-e2a9-4c1a-8e68-04382a7fff76", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"29aae689-111d-459f-9a4a-d1ad1d9d2761\",\r\n \"Name\": \"uipath\",\r\n \"ConnectorKey\": \"uipath-freshworks-freshdesk\",\r\n \"ConnectorName\": \"Freshdesk\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"No\",\r\n \"Folder\": \"uipath-rpa-isActivities\",\r\n \"FolderKey\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Created\": \"2026-06-18T13:24:02.950Z\",\r\n \"Updated\": \"2026-06-19T11:50:51.870Z\"\r\n },\r\n {\r\n \"Id\": \"989fd9d2-4b07-443e-86e3-ed6f1b00079b\",\r\n \"Name\": \"original_email@test.com\",\r\n \"ConnectorKey\": \"uipath-box-box\",\r\n \"ConnectorName\": \"Box\",\r\n \"State\": \"Failed\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"uipath-rpa-isActivities\",\r\n \"FolderKey\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Created\": \"2026-06-19T07:46:21.196Z\",\r\n \"Updated\": \"2026-06-19T12:08:33.426Z\"\r\n }\r\n ],\r\n \"Warning\": \"Connection(s) not enabled: 989fd9d2-4b07-443e-86e3-ed6f1b00079b. Only Enabled connections can be used for operations. Run 'uip is connections edit ' to re-authenticate.\"\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list", + "--all", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 3, + "stdout": "{\r\n \"Result\": \"ValidationError\",\r\n \"Message\": \"error: unknown option '--all'\",\r\n \"Instructions\": \"Check command arguments and options. Use --help for usage information.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "3baa335d-8cd1-44cf-936c-df8c013c99df", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"3baa335d-8cd1-44cf-936c-df8c013c99df\",\r\n \"StartTime\": \"2026-06-19T11:49:55.423Z\",\r\n \"EndTime\": \"2026-06-19T11:50:00.733Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"ResourceOverwrites\": \"{\\\"Connection.989fd9d2-4b07-443e-86e3-ed6f1b00079b\\\":{\\\"EntityDisplayName\\\":\\\"original_email@test.com\\\",\\\"Level\\\":0,\\\"Properties\\\":{\\\"connectionId\\\":\\\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\\\",\\\"folderKey\\\":\\\"b262689d-e2a9-4c1a-8e68-04382a7fff76\\\",\\\"ConnectionId\\\":\\\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\\\"}}}\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f730fadc-abfb-4c12-a627-915f8a99e224\",\r\n \"Info\": \"Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\r\\n\\r\\nin Main.xaml\\n at ConnectorActivity \\\"Create Folder\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-19T11:49:52.263Z\",\r\n \"ReleaseName\": \"TokenRefreshFailedJob\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"B6818817-9380-4F00-6A5A-0B49C962BBA9\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Serverless\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1089281,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 2979245,\r\n \"FolderKey\": \"eba26f86-ea3a-4955-a029-28c6e5006dab\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": true,\r\n \"LocalSystemAccount\": \"ROBOTUSER\",\r\n \"OrchestratorUserIdentity\": \"replacement_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"ServerlessJobType\": \"RobotJob\",\r\n \"LastModificationTime\": \"2026-06-19T11:50:00.733Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"3baa335d8cd144cf936cdf8c013c99df\",\r\n \"Id\": 50250322\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "history", + "3baa335d-8cd1-44cf-936c-df8c013c99df", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-19T11:50:00.760Z\",\r\n \"JobId\": 50250322\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-19T11:49:55.433Z\",\r\n \"JobId\": 50250322\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-19T11:49:52.307Z\",\r\n \"JobId\": 50250322\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "3baa335d-8cd1-44cf-936c-df8c013c99df", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-856b-c1dd9e521e15\",\r\n \"TraceId\": \"3baa335d-8cd1-44cf-936c-df8c013c99df\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 3baa335d-8cd1-44cf-936c-df8c013c99df\",\r\n \"StartTime\": \"2026-06-19T11:49:58.2096486Z\",\r\n \"EndTime\": \"2026-06-19T11:50:00.5096756Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Connection is disabled. Please enable the connection to continue. Error code: DAP-GE-3005.\\\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"311c6049-4966-4ef2-a6f3-8ceccaf38d54\",\r\n \"TenantId\": \"69d43765-6ede-4575-88b7-9db601a2c056\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"eba26f86-ea3a-4955-a029-28c6e5006dab\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"f9ecba5f-8dd8-4505-aeee-76ec5777d954\",\r\n \"JobKey\": \"3baa335d-8cd1-44cf-936c-df8c013c99df\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-19T11:50:00.7904098Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "989fd9d2-4b07-443e-86e3-ed6f1b00079b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Code\": \"ConnectionNotEnabled\",\r\n \"Message\": \"Connection '989fd9d2-4b07-443e-86e3-ed6f1b00079b' is not enabled (status: Failed).\",\r\n \"Instructions\": \"Run 'uip is connections edit ' to re-authenticate, or 'uip is connections create ' to create a new connection.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"TokenRefreshFailed_DAPRuntime\",\r\n \"ID\": 2979245,\r\n \"Key\": \"eba26f86-ea3a-4955-a029-28c6e5006dab\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-b4b03014\",\r\n \"ID\": 2978732,\r\n \"Key\": \"771ec4f1-f031-463e-a6b5-5ebfec25ba86\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-b4b03014\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-deploy-folder-4ab5f95e\",\r\n \"ID\": 2976398,\r\n \"Key\": \"59a72d90-70b6-4272-a665-325b0eaf9d62\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-deploy-folder-4ab5f95e\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-e5b01329\",\r\n \"ID\": 2976354,\r\n \"Key\": \"c2bba2f7-5f5f-4bfd-9781-ff7b5e11efef\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-e5b01329\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-8c88e3dc-B\",\r\n \"ID\": 2976154,\r\n \"Key\": \"cacc23b2-b067-4191-9ea1-b73b3b2e98a0\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-8c88e3dc-A\",\r\n \"ID\": 2976153,\r\n \"Key\": \"10908d6a-f2a8-4296-a034-d91137aee4cb\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-8c88e3dc-top\",\r\n \"ID\": 2976151,\r\n \"Key\": \"a027aeca-4f5d-4fb3-92f8-b8e97619b595\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-72c44bd7\",\r\n \"ID\": 2976049,\r\n \"Key\": \"fb61e5ba-40ef-4aca-9830-0a86dd6cfdae\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-72c44bd7\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"RequestFailed_DAPRuntime\",\r\n \"ID\": 2976014,\r\n \"Key\": \"e555fecd-20d3-47aa-864c-1c2e588d4769\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"uipath-rpa-isActivities\",\r\n \"ID\": 2975935,\r\n \"Key\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-B\",\r\n \"ID\": 2975612,\r\n \"Key\": \"94f779c1-f602-49d1-a253-8a6bc18c0c45\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-A\",\r\n \"ID\": 2975609,\r\n \"Key\": \"fb57fea0-6855-4bd2-8b86-3bc9332a545f\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-top\",\r\n \"ID\": 2975605,\r\n \"Key\": \"7f121d04-2efb-4a6e-964a-2b00c7ac41e5\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-5b56c68d\",\r\n \"ID\": 2975488,\r\n \"Key\": \"3cbad2ee-ba8a-4d87-8048-985d7bb95336\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-5b56c68d\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-00fb23da\",\r\n \"ID\": 2975403,\r\n \"Key\": \"95d78e99-e937-417f-8e0c-82d6e46c04bd\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-00fb23da\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-22d1b6dd\",\r\n \"ID\": 2974296,\r\n \"Key\": \"2292cfbc-81ef-4252-a991-554de4bf2d48\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-22d1b6dd\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-acdd13ce\",\r\n \"ID\": 2974264,\r\n \"Key\": \"e1e0b094-425b-4a36-a7ec-0fe528e6ac1c\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-acdd13ce\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"CaseTest\",\r\n \"ID\": 2973309,\r\n \"Key\": \"583f75bc-3630-4d55-b269-5c8af1f17b80\",\r\n \"Path\": \"Shared/uipath-maestro-case/CaseTest\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-deploy-folder-7188808a\",\r\n \"ID\": 2973247,\r\n \"Key\": \"40d377a0-0938-4e07-ad53-3e4b1f4dccdb\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-deploy-folder-7188808a\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-d6910c88\",\r\n \"ID\": 2973163,\r\n \"Key\": \"c1ddde01-465d-4684-a603-b2b96d87af94\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-d6910c88\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-0875838e\",\r\n \"ID\": 2973103,\r\n \"Key\": \"9cabacb6-8568-46e8-8cca-21a46f1a236b\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-0875838e\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-B\",\r\n \"ID\": 2973045,\r\n \"Key\": \"a909fdc9-4d95-48ab-9cf3-1f1c7e60d5de\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-A\",\r\n \"ID\": 2973044,\r\n \"Key\": \"c14bac5f-e6f0-4cc1-bb72-486f229421d2\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-top\",\r\n \"ID\": 2973043,\r\n \"Key\": \"37fc9a90-9d49-414a-9158-c883eb0cd43b\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-B\",\r\n \"ID\": 2955039,\r\n \"Key\": \"e66dbaa5-d7cf-4b9e-b66c-d42b372a2df1\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-A\",\r\n \"ID\": 2955037,\r\n \"Key\": \"46cc134a-e616-47ec-9e9a-360bf905ed5d\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-top\",\r\n \"ID\": 2955036,\r\n \"Key\": \"d6cd4cc4-a00e-4b87-8c46-4044385ddb3f\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-8df5d996\",\r\n \"ID\": 2954999,\r\n \"Key\": \"cd615438-aad8-4efe-af56-557bad7c1c2c\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-8df5d996\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-fce8a341\",\r\n \"ID\": 2954890,\r\n \"Key\": \"ad9bf36f-32e3-42b8-b051-c6cb114902ea\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-fce8a341\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset scoping isolation check\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-B\",\r\n \"ID\": 2952698,\r\n \"Key\": \"a1ef65e1-0213-4b50-ac52-e00b8235cad9\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-A\",\r\n \"ID\": 2952696,\r\n \"Key\": \"7a03267f-ec72-4cf6-a7a0-aa9128d8389a\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-top\",\r\n \"ID\": 2952694,\r\n \"Key\": \"c6f01161-4e94-4c59-a85a-76ac9b14963d\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-372553a0\",\r\n \"ID\": 2952623,\r\n \"Key\": \"5acebb10-f8d0-4c9a-a732-651f0f77a6ac\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-372553a0\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-e7972fc2\",\r\n \"ID\": 2952381,\r\n \"Key\": \"0347966b-d3db-4da4-b4ce-055d1b9a70a7\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-e7972fc2\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Solution 7\",\r\n \"ID\": 2933058,\r\n \"Key\": \"2c3f72f3-aaad-4cf6-b1bc-9e6504a3cb04\",\r\n \"Path\": \"Shared/Solution 7\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-dr-debug\",\r\n \"ID\": 2923711,\r\n \"Key\": \"993f1502-37ff-43f4-81d9-3c6b040d8bbf\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-dr-debug\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-dailyreport\",\r\n \"ID\": 2923530,\r\n \"Key\": \"1c8e5662-0616-410f-b9e7-c01003b467c2\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-dailyreport\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"ExampleV3\",\r\n \"ID\": 2923525,\r\n \"Key\": \"1fc080e4-a221-44fe-a9b5-9f526637502f\",\r\n \"Path\": \"Shared/ExampleV3\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-debug\",\r\n \"ID\": 2923476,\r\n \"Key\": \"26315131-b943-4d72-9442-3476a0196bec\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-debug\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"FraudEscalation\",\r\n \"ID\": 2922399,\r\n \"Key\": \"8492394e-2f77-4b7d-96c1-c0b0662e3819\",\r\n \"Path\": \"Shared/uipath-agents/FraudEscalation\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-B\",\r\n \"ID\": 2901931,\r\n \"Key\": \"fefa7912-d5d3-4dd5-87ea-3c4e3813321f\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-A\",\r\n \"ID\": 2901928,\r\n \"Key\": \"44fd0e78-9eda-4a45-84f5-b2e07473f312\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-top\",\r\n \"ID\": 2901927,\r\n \"Key\": \"ffd6cb46-e6b8-4c58-94b4-3ebd480903c0\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Finance\",\r\n \"ID\": 2901921,\r\n \"Key\": \"fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed\",\r\n \"Path\": \"Finance\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-0a0a527c\",\r\n \"ID\": 2901912,\r\n \"Key\": \"c8c3f842-ef1f-4e9a-9728-191cba4bfd89\",\r\n \"Path\": \"Shared/e2e-folder-a-0a0a527c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-0a0a527c\",\r\n \"ID\": 2901910,\r\n \"Key\": \"0098b403-912d-48bf-983b-0bf7cc7a1240\",\r\n \"Path\": \"Shared/e2e-folder-b-0a0a527c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-8805f811\",\r\n \"ID\": 2901871,\r\n \"Key\": \"f573d08c-c8fa-485b-ad39-917b502eca45\",\r\n \"Path\": \"Shared/e2e-folder-a-8805f811\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder A for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-8805f811\",\r\n \"ID\": 2901870,\r\n \"Key\": \"66247ea0-e4ad-4eb5-989d-846926c0acd5\",\r\n \"Path\": \"Shared/e2e-folder-b-8805f811\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-a441329c\",\r\n \"ID\": 2901830,\r\n \"Key\": \"b5eb7093-74cf-443c-8d02-8712f4831c5a\",\r\n \"Path\": \"Shared/e2e-folder-b-a441329c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-a441329c\",\r\n \"ID\": 2901829,\r\n \"Key\": \"616562ea-0b92-41bc-9c3e-110c3981a647\",\r\n \"Path\": \"Shared/e2e-folder-a-a441329c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder A for asset sharing\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/task.yaml index 016bab4d3b..6ad8bfd32b 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-disabled-dap/task.yaml @@ -15,21 +15,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/README.md new file mode 100644 index 0000000000..8cda813b4f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/README.md @@ -0,0 +1,68 @@ +# rpa-is-connection-missing-dap — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim `uip` CLI +responses captured from that session. + +## What the original session uncovered + +An Orchestrator unattended job (`TokenRefreshFailedJob`) in the Solution folder +`Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime` faulted with +Integration Service error **`DAP-GE-3000` (FailedToGetConnection)** at the +`ConnectorActivity "Create Folder"` in `Main.xaml`. The root cause: the **Box** +connection the process is bound to (`989fd9d2-…`, in folder `b262689d-…`) had +been **deleted** — `ping` returns **HTTP 404**, the bound folder now contains +only the Freshdesk connection, and no Box connection exists tenant-wide. Because +the published process still references the deleted connection ID, IS cannot +resolve it. + +This is the *deleted-connection* variant of the connection-not-resolved family — +distinct from a *disabled* connection (`DAP-GE-3005`): there the connection still +exists and can be re-authenticated; here it is gone, so the fix is to **create a +new connection** and **re-bind + republish** the process. (See the sibling +scenario `rpa-is-connection-disabled-dap` for the `DAP-GE-3005` case.) Per the +ground truth, after creating the new Box connection and re-binding, the re-run +completes Successfully. See `RESOLUTION.md`. + +The test exercises the agent's ability to (a) trust the emitted DAP code over the +folder name, (b) recover the connection ID from the job's `ResourceOverwrites`, +(c) distinguish *deleted* (404) from merely *disabled*, and (d) recommend +create-new + rebind rather than re-authenticate. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +This investigation was CLI-driven against a Studio Web project (no local source), +so there is **no** `process/` snapshot. + +> Note: state-dependent fixtures (`jobs list`, `is connections ping 989fd9d2-…`, +> the bound-folder `is connections list --folder-key b262689d-…`, and the +> tenant-wide `is connections list`) capture the **pre-fix** state at +> investigation time: connection `989fd9d2-…` deleted (404), no Box connection +> present, and `c3a00022-…` as the latest faulted job. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). +- An LLM judge grades the agent's final answer against `RESOLUTION.md`: correct + root cause (deleted/missing Box connection → `DAP-GE-3000`, not merely + disabled) **and** correct fix (create a new connection + re-bind + republish). + +## Re-running the extraction + +If the source transcript changes, regenerate the scaffold, then re-apply the +scenario-scoped manifest and the pre-fix state-dependent fixtures: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation <.local/investigations dir> --transcript \ + --resolution RESOLUTION.md --scenario-name rpa-is-connection-missing-dap --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/02e8acc99f.json deleted file mode 100644 index 6205d04188..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/081b84c7c0.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/081b84c7c0.json deleted file mode 100644 index 3d67262d3c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/081b84c7c0.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "Message", - "Data": { - "Message": "No connections found for any connector in folder 'eba26f86-ea3a-4955-a029-28c6e5006dab'." - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/177e41f489.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/177e41f489.json deleted file mode 100644 index 104b7eb374..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/177e41f489.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-1745-5e1c1bc6c1ba", - "TraceId": "c3a00022-67c7-4b04-8faa-35a726d641cc", - "ParentId": null, - "Name": "Robot Job c3a00022-67c7-4b04-8faa-35a726d641cc", - "StartTime": "2026-06-19T13:00:29.1938336Z", - "EndTime": "2026-06-19T13:00:31.2040699Z", - "Attributes": "{\"Exception\":\"UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass12_0.<b__0>d.MoveNext()\\n--- End of stack trace from previous location ---\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh)\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct)\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n --- End of inner exception stack trace ---\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"}", - "Status": 2, - "OrganizationId": "311c6049-4966-4ef2-a6f3-8ceccaf38d54", - "TenantId": "69d43765-6ede-4575-88b7-9db601a2c056", - "ExpiryTimeUtc": null, - "FolderKey": "eba26f86-ea3a-4955-a029-28c6e5006dab", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "f9ecba5f-8dd8-4505-aeee-76ec5777d954", - "JobKey": "c3a00022-67c7-4b04-8faa-35a726d641cc", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-06-19T13:00:31.5124967Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/2e37ebc77e.json deleted file mode 100644 index 44bfb742f1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,406 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "TokenRefreshFailed_DAPRuntime", - "ID": 2979245, - "Key": "eba26f86-ea3a-4955-a029-28c6e5006dab", - "Path": "Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-folder-b-b4b03014", - "ID": 2978732, - "Key": "771ec4f1-f031-463e-a6b5-5ebfec25ba86", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-b4b03014", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-deploy-folder-4ab5f95e", - "ID": 2976398, - "Key": "59a72d90-70b6-4272-a665-325b0eaf9d62", - "Path": "Shared/uipath-platform-e2e/e2e-deploy-folder-4ab5f95e", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-e5b01329", - "ID": 2976354, - "Key": "c2bba2f7-5f5f-4bfd-9781-ff7b5e11efef", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-e5b01329", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-8c88e3dc-B", - "ID": 2976154, - "Key": "cacc23b2-b067-4191-9ea1-b73b3b2e98a0", - "Path": "Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-8c88e3dc-A", - "ID": 2976153, - "Key": "10908d6a-f2a8-4296-a034-d91137aee4cb", - "Path": "Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-8c88e3dc-top", - "ID": 2976151, - "Key": "a027aeca-4f5d-4fb3-92f8-b8e97619b595", - "Path": "Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-72c44bd7", - "ID": 2976049, - "Key": "fb61e5ba-40ef-4aca-9830-0a86dd6cfdae", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-72c44bd7", - "Type": "Standard", - "Description": "" - }, - { - "Name": "RequestFailed_DAPRuntime", - "ID": 2976014, - "Key": "e555fecd-20d3-47aa-864c-1c2e588d4769", - "Path": "Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime", - "Type": "Solution", - "Description": "" - }, - { - "Name": "uipath-rpa-isActivities", - "ID": 2975935, - "Key": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Path": "Shared/uipath-rpa-isActivities", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-B", - "ID": 2975612, - "Key": "94f779c1-f602-49d1-a253-8a6bc18c0c45", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-A", - "ID": 2975609, - "Key": "fb57fea0-6855-4bd2-8b86-3bc9332a545f", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-top", - "ID": 2975605, - "Key": "7f121d04-2efb-4a6e-964a-2b00c7ac41e5", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-5b56c68d", - "ID": 2975488, - "Key": "3cbad2ee-ba8a-4d87-8048-985d7bb95336", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-5b56c68d", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-00fb23da", - "ID": 2975403, - "Key": "95d78e99-e937-417f-8e0c-82d6e46c04bd", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-00fb23da", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-22d1b6dd", - "ID": 2974296, - "Key": "2292cfbc-81ef-4252-a991-554de4bf2d48", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-22d1b6dd", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-acdd13ce", - "ID": 2974264, - "Key": "e1e0b094-425b-4a36-a7ec-0fe528e6ac1c", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-acdd13ce", - "Type": "Standard", - "Description": "" - }, - { - "Name": "CaseTest", - "ID": 2973309, - "Key": "583f75bc-3630-4d55-b269-5c8af1f17b80", - "Path": "Shared/uipath-maestro-case/CaseTest", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-deploy-folder-7188808a", - "ID": 2973247, - "Key": "40d377a0-0938-4e07-ad53-3e4b1f4dccdb", - "Path": "Shared/uipath-platform-e2e/e2e-deploy-folder-7188808a", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-folder-b-d6910c88", - "ID": 2973163, - "Key": "c1ddde01-465d-4684-a603-b2b96d87af94", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-d6910c88", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-0875838e", - "ID": 2973103, - "Key": "9cabacb6-8568-46e8-8cca-21a46f1a236b", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-0875838e", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-B", - "ID": 2973045, - "Key": "a909fdc9-4d95-48ab-9cf3-1f1c7e60d5de", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-A", - "ID": 2973044, - "Key": "c14bac5f-e6f0-4cc1-bb72-486f229421d2", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-top", - "ID": 2973043, - "Key": "37fc9a90-9d49-414a-9158-c883eb0cd43b", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-B", - "ID": 2955039, - "Key": "e66dbaa5-d7cf-4b9e-b66c-d42b372a2df1", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-A", - "ID": 2955037, - "Key": "46cc134a-e616-47ec-9e9a-360bf905ed5d", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-top", - "ID": 2955036, - "Key": "d6cd4cc4-a00e-4b87-8c46-4044385ddb3f", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-8df5d996", - "ID": 2954999, - "Key": "cd615438-aad8-4efe-af56-557bad7c1c2c", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-8df5d996", - "Type": "Standard", - "Description": "E2E test folder B for asset sharing" - }, - { - "Name": "e2e-folder-b-fce8a341", - "ID": 2954890, - "Key": "ad9bf36f-32e3-42b8-b051-c6cb114902ea", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-fce8a341", - "Type": "Standard", - "Description": "E2E test folder B for asset scoping isolation check" - }, - { - "Name": "e2e-fh-0b53372d-B", - "ID": 2952698, - "Key": "a1ef65e1-0213-4b50-ac52-e00b8235cad9", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-0b53372d-A", - "ID": 2952696, - "Key": "7a03267f-ec72-4cf6-a7a0-aa9128d8389a", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-0b53372d-top", - "ID": 2952694, - "Key": "c6f01161-4e94-4c59-a85a-76ac9b14963d", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-372553a0", - "ID": 2952623, - "Key": "5acebb10-f8d0-4c9a-a732-651f0f77a6ac", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-372553a0", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-e7972fc2", - "ID": 2952381, - "Key": "0347966b-d3db-4da4-b4ce-055d1b9a70a7", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-e7972fc2", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Solution 7", - "ID": 2933058, - "Key": "2c3f72f3-aaad-4cf6-b1bc-9e6504a3cb04", - "Path": "Shared/Solution 7", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-dr-debug", - "ID": 2923711, - "Key": "993f1502-37ff-43f4-81d9-3c6b040d8bbf", - "Path": "Shared/uipath-maestro-flow/Example-dr-debug", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-dailyreport", - "ID": 2923530, - "Key": "1c8e5662-0616-410f-b9e7-c01003b467c2", - "Path": "Shared/uipath-maestro-flow/Example-dailyreport", - "Type": "Solution", - "Description": "" - }, - { - "Name": "ExampleV3", - "ID": 2923525, - "Key": "1fc080e4-a221-44fe-a9b5-9f526637502f", - "Path": "Shared/ExampleV3", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-debug", - "ID": 2923476, - "Key": "26315131-b943-4d72-9442-3476a0196bec", - "Path": "Shared/uipath-maestro-flow/Example-debug", - "Type": "Solution", - "Description": "" - }, - { - "Name": "FraudEscalation", - "ID": 2922399, - "Key": "8492394e-2f77-4b7d-96c1-c0b0662e3819", - "Path": "Shared/uipath-agents/FraudEscalation", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-B", - "ID": 2901931, - "Key": "fefa7912-d5d3-4dd5-87ea-3c4e3813321f", - "Path": "Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-A", - "ID": 2901928, - "Key": "44fd0e78-9eda-4a45-84f5-b2e07473f312", - "Path": "Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-top", - "ID": 2901927, - "Key": "ffd6cb46-e6b8-4c58-94b4-3ebd480903c0", - "Path": "Shared/e2e-fh-eecfa769-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Finance", - "ID": 2901921, - "Key": "fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed", - "Path": "Finance", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-a-0a0a527c", - "ID": 2901912, - "Key": "c8c3f842-ef1f-4e9a-9728-191cba4bfd89", - "Path": "Shared/e2e-folder-a-0a0a527c", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-0a0a527c", - "ID": 2901910, - "Key": "0098b403-912d-48bf-983b-0bf7cc7a1240", - "Path": "Shared/e2e-folder-b-0a0a527c", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-a-8805f811", - "ID": 2901871, - "Key": "f573d08c-c8fa-485b-ad39-917b502eca45", - "Path": "Shared/e2e-folder-a-8805f811", - "Type": "Standard", - "Description": "E2E test folder A for asset sharing" - }, - { - "Name": "e2e-folder-b-8805f811", - "ID": 2901870, - "Key": "66247ea0-e4ad-4eb5-989d-846926c0acd5", - "Path": "Shared/e2e-folder-b-8805f811", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-a441329c", - "ID": 2901830, - "Key": "b5eb7093-74cf-443c-8d02-8712f4831c5a", - "Path": "Shared/e2e-folder-b-a441329c", - "Type": "Standard", - "Description": "E2E test folder B for asset sharing" - }, - { - "Name": "e2e-folder-a-a441329c", - "ID": 2901829, - "Key": "616562ea-0b92-41bc-9c3e-110c3981a647", - "Path": "Shared/e2e-folder-a-a441329c", - "Type": "Standard", - "Description": "E2E test folder A for asset sharing" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/4dc2cb914e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/4dc2cb914e.json deleted file mode 100644 index 984ffa0014..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/4dc2cb914e.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "HTTP 404: Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it", - "Instructions": "Run 'uip is connections edit ' to re-authenticate, or 'uip is connections create ' to create a new connection." -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/5d253e87ea.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/5d253e87ea.json deleted file mode 100644 index 914444bc5a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/5d253e87ea.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3a00022-67c7-4b04-8faa-35a726d641cc", - "StartTime": "2026-06-19T13:00:25.170Z", - "EndTime": "2026-06-19T13:00:31.470Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "ResourceOverwrites": "{\"Connection.989fd9d2-4b07-443e-86e3-ed6f1b00079b\":{\"EntityDisplayName\":\"original_email@test.com\",\"Level\":0,\"Properties\":{\"connectionId\":\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\",\"folderKey\":\"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\"ConnectionId\":\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\"}}}", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "cba954c8-212c-4741-b3ac-1ac85b6436d4", - "Info": "Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\r\n\r\nin Main.xaml\n at ConnectorActivity \"Create Folder\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass12_0.<b__0>d.MoveNext()\n--- End of stack trace from previous location ---\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh)\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct)\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\n --- End of inner exception stack trace ---\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-19T13:00:22.713Z", - "ReleaseName": "TokenRefreshFailedJob", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "6CC4E15E-EB81-4F00-544B-0B49C96C80C6", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Serverless", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1089281, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 2979245, - "FolderKey": "eba26f86-ea3a-4955-a029-28c6e5006dab", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": true, - "LocalSystemAccount": "ROBOTUSER", - "OrchestratorUserIdentity": "replacement_email@test.com", - "RemoteControlAccess": "None", - "ServerlessJobType": "RobotJob", - "LastModificationTime": "2026-06-19T13:00:31.470Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3a0002267c74b048faa35a726d641cc", - "Id": 50252766 - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/74f92d894a.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/74f92d894a.json deleted file mode 100644 index f5ae4360e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/74f92d894a.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-19T13:00:31.177Z", - "Message": "Create Folder: Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\nin Main.xaml\n at ConnectorActivity \"Create Folder\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/8a280b15db.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/8a280b15db.json deleted file mode 100644 index a44bbd5014..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/8a280b15db.json +++ /dev/null @@ -1,110 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "e829d77c-2d88-472a-a534-81459997fcfd", - "Name": "user3@test.com", - "ConnectorKey": "uipath-microsoft-teams", - "ConnectorName": "Microsoft Teams", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-05-13T10:52:13.873Z", - "Updated": "2026-06-19T07:46:58.243Z" - }, - { - "Id": "edaf1fb3-e787-4184-946f-cc0c378f156e", - "Name": "user5@test.com", - "ConnectorKey": "uipath-zendesk-zendesk", - "ConnectorName": "Zendesk", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-05-13T11:03:12.636Z", - "Updated": "2026-06-19T07:46:56.843Z" - }, - { - "Id": "0b78b83b-795d-49a2-951a-19306249f26b", - "Name": "dev225223", - "ConnectorKey": "uipath-servicenow-servicenow", - "ConnectorName": "ServiceNow", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-05-13T11:07:09.220Z", - "Updated": "2026-06-19T07:46:57.800Z" - }, - { - "Id": "1ff8b174-8f76-4249-a467-7846f10dd6b7", - "Name": "user6@test.com-uipath-sandbox-380", - "ConnectorKey": "uipath-atlassian-jira", - "ConnectorName": "Jira", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-05-13T11:08:27.966Z", - "Updated": "2026-06-19T07:46:57.516Z" - }, - { - "Id": "9d95be15-8820-466e-9d87-745f23e56d9f", - "Name": "2144368", - "ConnectorKey": "uipath-oracle-netsuite", - "ConnectorName": "Oracle NetSuite", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-06-15T11:26:33.420Z", - "Updated": "2026-06-19T07:46:57.283Z" - }, - { - "Id": "786ad982-8263-4ba2-941e-38dae2f5c8a8", - "Name": "is-sandboxes", - "ConnectorKey": "uipath-salesforce-slack", - "ConnectorName": "Slack", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-06-17T07:01:22.243Z", - "Updated": "2026-06-19T07:46:57.783Z" - }, - { - "Id": "ec84c4af-b354-4209-b388-5fbdb98ee800", - "Name": "user5@test.com", - "ConnectorKey": "uipath-citrix-sharefile", - "ConnectorName": "Citrix ShareFile", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-06-17T07:08:04.443Z", - "Updated": "2026-06-19T07:46:57.853Z" - }, - { - "Id": "65a6d5e5-7250-4a98-a7b9-931965b5ce4e", - "Name": "uipath", - "ConnectorKey": "uipath-freshworks-freshdesk", - "ConnectorName": "Freshdesk", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "Yes", - "Folder": "replacement_email@test.com's workspace", - "FolderKey": "acdfea6d-632a-4327-94b3-e050717448d8", - "Created": "2026-06-18T11:35:25.686Z", - "Updated": "2026-06-19T07:46:57.366Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/README.md deleted file mode 100644 index c0181bf713..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/README.md +++ /dev/null @@ -1,67 +0,0 @@ -# rpa-is-connection-missing-dap — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim `uip` CLI -responses captured from that session. - -## What the original session uncovered - -An Orchestrator unattended job (`TokenRefreshFailedJob`) in the Solution folder -`Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime` faulted with -Integration Service error **`DAP-GE-3000` (FailedToGetConnection)** at the -`ConnectorActivity "Create Folder"` in `Main.xaml`. The root cause: the **Box** -connection the process is bound to (`989fd9d2-…`, in folder `b262689d-…`) had -been **deleted** — `ping` returns **HTTP 404**, the bound folder now contains -only the Freshdesk connection, and no Box connection exists tenant-wide. Because -the published process still references the deleted connection ID, IS cannot -resolve it. - -This is the *deleted-connection* variant of the connection-not-resolved family — -distinct from a *disabled* connection (`DAP-GE-3005`): there the connection still -exists and can be re-authenticated; here it is gone, so the fix is to **create a -new connection** and **re-bind + republish** the process. (See the sibling -scenario `rpa-is-connection-disabled-dap` for the `DAP-GE-3005` case.) Per the -ground truth, after creating the new Box connection and re-binding, the re-run -completes Successfully. See `RESOLUTION.md`. - -The test exercises the agent's ability to (a) trust the emitted DAP code over the -folder name, (b) recover the connection ID from the job's `ResourceOverwrites`, -(c) distinguish *deleted* (404) from merely *disabled*, and (d) recommend -create-new + rebind rather than re-authenticate. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real `uip` stdout captured verbatim from the session | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -This investigation was CLI-driven against a Studio Web project (no local source), -so there is **no** `process/` snapshot. - -> Note: state-dependent fixtures (`jobs list`, `is connections ping 989fd9d2-…`, -> the bound-folder `is connections list --folder-key b262689d-…`, and the -> tenant-wide `is connections list`) capture the **pre-fix** state at -> investigation time: connection `989fd9d2-…` deleted (404), no Box connection -> present, and `c3a00022-…` as the latest faulted job. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill (`skill_triggered`). -- An LLM judge grades the agent's final answer against `RESOLUTION.md`: correct - root cause (deleted/missing Box connection → `DAP-GE-3000`, not merely - disabled) **and** correct fix (create a new connection + re-bind + republish). - -## Re-running the extraction - -If the source transcript changes, regenerate the scaffold, then re-apply the -scenario-scoped manifest and the pre-fix state-dependent fixtures: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation <.local/investigations dir> --transcript \ - --resolution RESOLUTION.md --scenario-name rpa-is-connection-missing-dap --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/d30940f343.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/d30940f343.json deleted file mode 100644 index 7f31a56d2c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/d30940f343.json +++ /dev/null @@ -1,77 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c3a00022-67c7-4b04-8faa-35a726d641cc", - "State": "Faulted", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "6CC4E15E-EB81-4F00-544B-0B49C96C80C6", - "StartTime": "2026-06-19T13:00:25.170Z", - "EndTime": "2026-06-19T13:00:31.470Z", - "CreationTime": "2026-06-19T13:00:22.713Z", - "Id": 50252766 - }, - { - "Key": "2524b63d-0730-4feb-9e5a-fc9a7f7a23f1", - "State": "Successful", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "C1ED99D1-5381-4F00-C7BD-0B49C96798C9", - "StartTime": "2026-06-19T12:21:56.340Z", - "EndTime": "2026-06-19T12:22:02.817Z", - "CreationTime": "2026-06-19T12:21:53.570Z", - "Id": 50251460 - }, - { - "Key": "3baa335d-8cd1-44cf-936c-df8c013c99df", - "State": "Faulted", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "B6818817-9380-4F00-6A5A-0B49C962BBA9", - "StartTime": "2026-06-19T11:49:55.423Z", - "EndTime": "2026-06-19T11:50:00.733Z", - "CreationTime": "2026-06-19T11:49:52.263Z", - "Id": 50250322 - }, - { - "Key": "0a2e8a22-ccb4-4953-9356-151d32a7e851", - "State": "Faulted", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "8A852950-F07E-4F00-AC73-0B49C95A9A71", - "StartTime": "2026-06-19T10:51:37.240Z", - "EndTime": "2026-06-19T10:51:45.237Z", - "CreationTime": "2026-06-19T10:51:32.910Z", - "Id": 50248422 - }, - { - "Key": "2c0c44a4-364c-4cd8-934d-7c73bb3ff354", - "State": "Successful", - "ProcessName": "TokenRefreshFailedJob", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "15117404-EA7A-4F00-444C-0B49C94419E8", - "StartTime": "2026-06-19T08:07:56.780Z", - "EndTime": "2026-06-19T08:08:03.683Z", - "CreationTime": "2026-06-19T08:07:45.107Z", - "Id": 50242038 - } - ], - "Pagination": { - "Returned": 5, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/dc664aaf9f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/dc664aaf9f.json deleted file mode 100644 index d6358fc11d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/dc664aaf9f.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-19T13:00:31.477Z", - "JobId": 50252766 - }, - { - "State": "Running", - "Timestamp": "2026-06-19T13:00:25.177Z", - "JobId": 50252766 - }, - { - "State": "Pending", - "Timestamp": "2026-06-19T13:00:22.780Z", - "JobId": 50252766 - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/e919f3989b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/e919f3989b.json deleted file mode 100644 index ae247368cd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/e919f3989b.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "ValidationError", - "Message": "error: unknown option '--all'", - "Instructions": "Check command arguments and options. Use --help for usage information." -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/f50db666cc.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/f50db666cc.json deleted file mode 100644 index 606498dd84..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/f50db666cc.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "29aae689-111d-459f-9a4a-d1ad1d9d2761", - "Name": "uipath", - "ConnectorKey": "uipath-freshworks-freshdesk", - "ConnectorName": "Freshdesk", - "State": "Enabled", - "Owner": "replacement_email@test.com", - "IsDefault": "No", - "Folder": "uipath-rpa-isActivities", - "FolderKey": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Created": "2026-06-18T13:24:02.950Z", - "Updated": "2026-06-19T11:50:51.870Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/manifest.json deleted file mode 100644 index e2d0e3a0f3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/m/r/manifest.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "version": 2, - "_doc": "Faithful-replay dispatch table for the rpa-is-connection-missing-dap scenario (DAP-GE-3000, deleted Box connection). Fixtures are verbatim uip CLI responses captured at investigation time. State-dependent fixtures (jobs list, bound-folder list, tenant-wide list, ping 989fd9d2) reflect the PRE-FIX state: connection 989fd9d2 deleted (404), no Box connection in folder b262689d or tenant-wide, c3a00022 the latest faulted job.", - "rules": [ - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list --all --output json", - "file": "e919f3989b.json", - "exit_code": 3 - }, - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key eba26f86-ea3a-4955-a029-28c6e5006dab --output json", - "file": "d30940f343.json" - }, - { - "match": "or jobs get c3a00022-67c7-4b04-8faa-35a726d641cc --output json", - "file": "5d253e87ea.json" - }, - { - "match": "or jobs logs c3a00022-67c7-4b04-8faa-35a726d641cc --level Error --output json", - "file": "74f92d894a.json" - }, - { - "match": "or jobs history c3a00022-67c7-4b04-8faa-35a726d641cc --output json", - "file": "dc664aaf9f.json" - }, - { - "match": "or jobs traces c3a00022-67c7-4b04-8faa-35a726d641cc --output json", - "file": "177e41f489.json" - }, - { - "match": "is connections list --folder-key eba26f86-ea3a-4955-a029-28c6e5006dab --output json", - "file": "081b84c7c0.json" - }, - { - "match": "is connections list --folder-key b262689d-e2a9-4c1a-8e68-04382a7fff76 --output json", - "file": "f50db666cc.json" - }, - { - "match": "is connections ping 989fd9d2-4b07-443e-86e3-ed6f1b00079b --output json", - "file": "4dc2cb914e.json" - }, - { - "match": "is connections list --output json", - "file": "8a280b15db.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/uip-fixture.json new file mode 100644 index 0000000000..4daab11bb5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/data/uip-fixture.json @@ -0,0 +1,163 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "c3a00022-67c7-4b04-8faa-35a726d641cc", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-19T13:00:31.177Z\",\r\n \"Message\": \"Create Folder: Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\\nin Main.xaml\\n at ConnectorActivity \\\"Create Folder\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "eba26f86-ea3a-4955-a029-28c6e5006dab", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3a00022-67c7-4b04-8faa-35a726d641cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"6CC4E15E-EB81-4F00-544B-0B49C96C80C6\",\r\n \"StartTime\": \"2026-06-19T13:00:25.170Z\",\r\n \"EndTime\": \"2026-06-19T13:00:31.470Z\",\r\n \"CreationTime\": \"2026-06-19T13:00:22.713Z\",\r\n \"Id\": 50252766\r\n },\r\n {\r\n \"Key\": \"2524b63d-0730-4feb-9e5a-fc9a7f7a23f1\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"C1ED99D1-5381-4F00-C7BD-0B49C96798C9\",\r\n \"StartTime\": \"2026-06-19T12:21:56.340Z\",\r\n \"EndTime\": \"2026-06-19T12:22:02.817Z\",\r\n \"CreationTime\": \"2026-06-19T12:21:53.570Z\",\r\n \"Id\": 50251460\r\n },\r\n {\r\n \"Key\": \"3baa335d-8cd1-44cf-936c-df8c013c99df\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"B6818817-9380-4F00-6A5A-0B49C962BBA9\",\r\n \"StartTime\": \"2026-06-19T11:49:55.423Z\",\r\n \"EndTime\": \"2026-06-19T11:50:00.733Z\",\r\n \"CreationTime\": \"2026-06-19T11:49:52.263Z\",\r\n \"Id\": 50250322\r\n },\r\n {\r\n \"Key\": \"0a2e8a22-ccb4-4953-9356-151d32a7e851\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"8A852950-F07E-4F00-AC73-0B49C95A9A71\",\r\n \"StartTime\": \"2026-06-19T10:51:37.240Z\",\r\n \"EndTime\": \"2026-06-19T10:51:45.237Z\",\r\n \"CreationTime\": \"2026-06-19T10:51:32.910Z\",\r\n \"Id\": 50248422\r\n },\r\n {\r\n \"Key\": \"2c0c44a4-364c-4cd8-934d-7c73bb3ff354\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"TokenRefreshFailedJob\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"15117404-EA7A-4F00-444C-0B49C94419E8\",\r\n \"StartTime\": \"2026-06-19T08:07:56.780Z\",\r\n \"EndTime\": \"2026-06-19T08:08:03.683Z\",\r\n \"CreationTime\": \"2026-06-19T08:07:45.107Z\",\r\n \"Id\": 50242038\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 5,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "eba26f86-ea3a-4955-a029-28c6e5006dab", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Message\",\r\n \"Data\": {\r\n \"Message\": \"No connections found for any connector in folder 'eba26f86-ea3a-4955-a029-28c6e5006dab'.\"\r\n }\r\n}" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "b262689d-e2a9-4c1a-8e68-04382a7fff76", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"29aae689-111d-459f-9a4a-d1ad1d9d2761\",\r\n \"Name\": \"uipath\",\r\n \"ConnectorKey\": \"uipath-freshworks-freshdesk\",\r\n \"ConnectorName\": \"Freshdesk\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"No\",\r\n \"Folder\": \"uipath-rpa-isActivities\",\r\n \"FolderKey\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Created\": \"2026-06-18T13:24:02.950Z\",\r\n \"Updated\": \"2026-06-19T11:50:51.870Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--all", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 3, + "stdout": "{\r\n \"Result\": \"ValidationError\",\r\n \"Message\": \"error: unknown option '--all'\",\r\n \"Instructions\": \"Check command arguments and options. Use --help for usage information.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3a00022-67c7-4b04-8faa-35a726d641cc", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3a00022-67c7-4b04-8faa-35a726d641cc\",\r\n \"StartTime\": \"2026-06-19T13:00:25.170Z\",\r\n \"EndTime\": \"2026-06-19T13:00:31.470Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"ResourceOverwrites\": \"{\\\"Connection.989fd9d2-4b07-443e-86e3-ed6f1b00079b\\\":{\\\"EntityDisplayName\\\":\\\"original_email@test.com\\\",\\\"Level\\\":0,\\\"Properties\\\":{\\\"connectionId\\\":\\\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\\\",\\\"folderKey\\\":\\\"b262689d-e2a9-4c1a-8e68-04382a7fff76\\\",\\\"ConnectionId\\\":\\\"989fd9d2-4b07-443e-86e3-ed6f1b00079b\\\"}}}\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"cba954c8-212c-4741-b3ac-1ac85b6436d4\",\r\n \"Info\": \"Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\\r\\n\\r\\nin Main.xaml\\n at ConnectorActivity \\\"Create Folder\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass12_0.<b__0>d.MoveNext()\\n--- End of stack trace from previous location ---\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh)\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct)\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n --- End of inner exception stack trace ---\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-19T13:00:22.713Z\",\r\n \"ReleaseName\": \"TokenRefreshFailedJob\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"6CC4E15E-EB81-4F00-544B-0B49C96C80C6\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Serverless\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1089281,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 2979245,\r\n \"FolderKey\": \"eba26f86-ea3a-4955-a029-28c6e5006dab\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": true,\r\n \"LocalSystemAccount\": \"ROBOTUSER\",\r\n \"OrchestratorUserIdentity\": \"replacement_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"ServerlessJobType\": \"RobotJob\",\r\n \"LastModificationTime\": \"2026-06-19T13:00:31.470Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3a0002267c74b048faa35a726d641cc\",\r\n \"Id\": 50252766\r\n }\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "history", + "c3a00022-67c7-4b04-8faa-35a726d641cc", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-19T13:00:31.477Z\",\r\n \"JobId\": 50252766\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-19T13:00:25.177Z\",\r\n \"JobId\": 50252766\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-19T13:00:22.780Z\",\r\n \"JobId\": 50252766\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "c3a00022-67c7-4b04-8faa-35a726d641cc", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-1745-5e1c1bc6c1ba\",\r\n \"TraceId\": \"c3a00022-67c7-4b04-8faa-35a726d641cc\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job c3a00022-67c7-4b04-8faa-35a726d641cc\",\r\n \"StartTime\": \"2026-06-19T13:00:29.1938336Z\",\r\n \"EndTime\": \"2026-06-19T13:00:31.2040699Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"UiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException: Failed to retrieve connection. Consider using a different connection. - Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it Error code: DAP-GE-3000.\\\\n ---> UiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it\\\\n at UiPath.ConnectionClient.Helpers.HttpResponseMessageExtensions.EnsureConnectionStatusCodeAsync(HttpResponseMessage response, CancellationToken ct)\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync(Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\n at UiPath.ConnectionClient.Helpers.HttpConnectionServiceClient.RequestWithRetryAsync[T](Uri uri, HttpMethod method, String content, TimeSpan timeout, Dictionary`2 headers, CancellationToken ct)\\\\n at UiPath.ConnectionClient.ConnectionClient.<>c__DisplayClass12_0.<b__0>d.MoveNext()\\\\n--- End of stack trace from previous location ---\\\\n at UiPath.ConnectionClient.Helpers.CacheHelpers`1.GetValueFromCache(MemoryCache cache, String key, Func`1 refreshValueFunc, DateTimeOffset expiry, SemaphoreSlim cacheSemaphore, Boolean forceRefresh)\\\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh, Boolean includeConfig)\\\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct, Boolean forceRefresh)\\\\n at UiPath.ConnectionClient.ConnectionClient.GetConnectionAsync(String connectionId, CancellationToken ct)\\\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\\\n --- End of inner exception stack trace ---\\\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken token)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivityBase.BuildExecutionParametersAsync(AsyncCodeActivityContext context, RequestParameters requestParameters, CancellationToken token)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"311c6049-4966-4ef2-a6f3-8ceccaf38d54\",\r\n \"TenantId\": \"69d43765-6ede-4575-88b7-9db601a2c056\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"eba26f86-ea3a-4955-a029-28c6e5006dab\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"f9ecba5f-8dd8-4505-aeee-76ec5777d954\",\r\n \"JobKey\": \"c3a00022-67c7-4b04-8faa-35a726d641cc\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-06-19T13:00:31.5124967Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "is", + "connections", + "ping", + "989fd9d2-4b07-443e-86e3-ed6f1b00079b", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"HTTP 404: Connection [989fd9d2-4b07-443e-86e3-ed6f1b00079b] is invalid or you do not have access to it\",\r\n \"Instructions\": \"Run 'uip is connections edit ' to re-authenticate, or 'uip is connections create ' to create a new connection.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"TokenRefreshFailed_DAPRuntime\",\r\n \"ID\": 2979245,\r\n \"Key\": \"eba26f86-ea3a-4955-a029-28c6e5006dab\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities/TokenRefreshFailed_DAPRuntime\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-b4b03014\",\r\n \"ID\": 2978732,\r\n \"Key\": \"771ec4f1-f031-463e-a6b5-5ebfec25ba86\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-b4b03014\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-deploy-folder-4ab5f95e\",\r\n \"ID\": 2976398,\r\n \"Key\": \"59a72d90-70b6-4272-a665-325b0eaf9d62\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-deploy-folder-4ab5f95e\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-e5b01329\",\r\n \"ID\": 2976354,\r\n \"Key\": \"c2bba2f7-5f5f-4bfd-9781-ff7b5e11efef\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-e5b01329\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-8c88e3dc-B\",\r\n \"ID\": 2976154,\r\n \"Key\": \"cacc23b2-b067-4191-9ea1-b73b3b2e98a0\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-8c88e3dc-A\",\r\n \"ID\": 2976153,\r\n \"Key\": \"10908d6a-f2a8-4296-a034-d91137aee4cb\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top/e2e-fh-8c88e3dc-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-8c88e3dc-top\",\r\n \"ID\": 2976151,\r\n \"Key\": \"a027aeca-4f5d-4fb3-92f8-b8e97619b595\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-8c88e3dc-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-72c44bd7\",\r\n \"ID\": 2976049,\r\n \"Key\": \"fb61e5ba-40ef-4aca-9830-0a86dd6cfdae\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-72c44bd7\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"RequestFailed_DAPRuntime\",\r\n \"ID\": 2976014,\r\n \"Key\": \"e555fecd-20d3-47aa-864c-1c2e588d4769\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"uipath-rpa-isActivities\",\r\n \"ID\": 2975935,\r\n \"Key\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-B\",\r\n \"ID\": 2975612,\r\n \"Key\": \"94f779c1-f602-49d1-a253-8a6bc18c0c45\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-A\",\r\n \"ID\": 2975609,\r\n \"Key\": \"fb57fea0-6855-4bd2-8b86-3bc9332a545f\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-top\",\r\n \"ID\": 2975605,\r\n \"Key\": \"7f121d04-2efb-4a6e-964a-2b00c7ac41e5\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-5b56c68d\",\r\n \"ID\": 2975488,\r\n \"Key\": \"3cbad2ee-ba8a-4d87-8048-985d7bb95336\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-5b56c68d\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-00fb23da\",\r\n \"ID\": 2975403,\r\n \"Key\": \"95d78e99-e937-417f-8e0c-82d6e46c04bd\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-00fb23da\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-22d1b6dd\",\r\n \"ID\": 2974296,\r\n \"Key\": \"2292cfbc-81ef-4252-a991-554de4bf2d48\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-22d1b6dd\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-acdd13ce\",\r\n \"ID\": 2974264,\r\n \"Key\": \"e1e0b094-425b-4a36-a7ec-0fe528e6ac1c\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-acdd13ce\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"CaseTest\",\r\n \"ID\": 2973309,\r\n \"Key\": \"583f75bc-3630-4d55-b269-5c8af1f17b80\",\r\n \"Path\": \"Shared/uipath-maestro-case/CaseTest\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-deploy-folder-7188808a\",\r\n \"ID\": 2973247,\r\n \"Key\": \"40d377a0-0938-4e07-ad53-3e4b1f4dccdb\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-deploy-folder-7188808a\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-d6910c88\",\r\n \"ID\": 2973163,\r\n \"Key\": \"c1ddde01-465d-4684-a603-b2b96d87af94\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-d6910c88\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-0875838e\",\r\n \"ID\": 2973103,\r\n \"Key\": \"9cabacb6-8568-46e8-8cca-21a46f1a236b\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-0875838e\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-B\",\r\n \"ID\": 2973045,\r\n \"Key\": \"a909fdc9-4d95-48ab-9cf3-1f1c7e60d5de\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-A\",\r\n \"ID\": 2973044,\r\n \"Key\": \"c14bac5f-e6f0-4cc1-bb72-486f229421d2\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-top\",\r\n \"ID\": 2973043,\r\n \"Key\": \"37fc9a90-9d49-414a-9158-c883eb0cd43b\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-B\",\r\n \"ID\": 2955039,\r\n \"Key\": \"e66dbaa5-d7cf-4b9e-b66c-d42b372a2df1\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-A\",\r\n \"ID\": 2955037,\r\n \"Key\": \"46cc134a-e616-47ec-9e9a-360bf905ed5d\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-top\",\r\n \"ID\": 2955036,\r\n \"Key\": \"d6cd4cc4-a00e-4b87-8c46-4044385ddb3f\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-8df5d996\",\r\n \"ID\": 2954999,\r\n \"Key\": \"cd615438-aad8-4efe-af56-557bad7c1c2c\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-8df5d996\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-fce8a341\",\r\n \"ID\": 2954890,\r\n \"Key\": \"ad9bf36f-32e3-42b8-b051-c6cb114902ea\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-fce8a341\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset scoping isolation check\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-B\",\r\n \"ID\": 2952698,\r\n \"Key\": \"a1ef65e1-0213-4b50-ac52-e00b8235cad9\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-A\",\r\n \"ID\": 2952696,\r\n \"Key\": \"7a03267f-ec72-4cf6-a7a0-aa9128d8389a\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-top\",\r\n \"ID\": 2952694,\r\n \"Key\": \"c6f01161-4e94-4c59-a85a-76ac9b14963d\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-372553a0\",\r\n \"ID\": 2952623,\r\n \"Key\": \"5acebb10-f8d0-4c9a-a732-651f0f77a6ac\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-372553a0\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-e7972fc2\",\r\n \"ID\": 2952381,\r\n \"Key\": \"0347966b-d3db-4da4-b4ce-055d1b9a70a7\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-e7972fc2\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Solution 7\",\r\n \"ID\": 2933058,\r\n \"Key\": \"2c3f72f3-aaad-4cf6-b1bc-9e6504a3cb04\",\r\n \"Path\": \"Shared/Solution 7\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-dr-debug\",\r\n \"ID\": 2923711,\r\n \"Key\": \"993f1502-37ff-43f4-81d9-3c6b040d8bbf\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-dr-debug\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-dailyreport\",\r\n \"ID\": 2923530,\r\n \"Key\": \"1c8e5662-0616-410f-b9e7-c01003b467c2\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-dailyreport\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"ExampleV3\",\r\n \"ID\": 2923525,\r\n \"Key\": \"1fc080e4-a221-44fe-a9b5-9f526637502f\",\r\n \"Path\": \"Shared/ExampleV3\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-debug\",\r\n \"ID\": 2923476,\r\n \"Key\": \"26315131-b943-4d72-9442-3476a0196bec\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-debug\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"FraudEscalation\",\r\n \"ID\": 2922399,\r\n \"Key\": \"8492394e-2f77-4b7d-96c1-c0b0662e3819\",\r\n \"Path\": \"Shared/uipath-agents/FraudEscalation\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-B\",\r\n \"ID\": 2901931,\r\n \"Key\": \"fefa7912-d5d3-4dd5-87ea-3c4e3813321f\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-A\",\r\n \"ID\": 2901928,\r\n \"Key\": \"44fd0e78-9eda-4a45-84f5-b2e07473f312\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-top\",\r\n \"ID\": 2901927,\r\n \"Key\": \"ffd6cb46-e6b8-4c58-94b4-3ebd480903c0\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Finance\",\r\n \"ID\": 2901921,\r\n \"Key\": \"fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed\",\r\n \"Path\": \"Finance\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-0a0a527c\",\r\n \"ID\": 2901912,\r\n \"Key\": \"c8c3f842-ef1f-4e9a-9728-191cba4bfd89\",\r\n \"Path\": \"Shared/e2e-folder-a-0a0a527c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-0a0a527c\",\r\n \"ID\": 2901910,\r\n \"Key\": \"0098b403-912d-48bf-983b-0bf7cc7a1240\",\r\n \"Path\": \"Shared/e2e-folder-b-0a0a527c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-8805f811\",\r\n \"ID\": 2901871,\r\n \"Key\": \"f573d08c-c8fa-485b-ad39-917b502eca45\",\r\n \"Path\": \"Shared/e2e-folder-a-8805f811\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder A for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-8805f811\",\r\n \"ID\": 2901870,\r\n \"Key\": \"66247ea0-e4ad-4eb5-989d-846926c0acd5\",\r\n \"Path\": \"Shared/e2e-folder-b-8805f811\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-a441329c\",\r\n \"ID\": 2901830,\r\n \"Key\": \"b5eb7093-74cf-443c-8d02-8712f4831c5a\",\r\n \"Path\": \"Shared/e2e-folder-b-a441329c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-a441329c\",\r\n \"ID\": 2901829,\r\n \"Key\": \"616562ea-0b92-41bc-9c3e-110c3981a647\",\r\n \"Path\": \"Shared/e2e-folder-a-a441329c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder A for asset sharing\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "is", + "connections", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"e829d77c-2d88-472a-a534-81459997fcfd\",\r\n \"Name\": \"user3@test.com\",\r\n \"ConnectorKey\": \"uipath-microsoft-teams\",\r\n \"ConnectorName\": \"Microsoft Teams\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-05-13T10:52:13.873Z\",\r\n \"Updated\": \"2026-06-19T07:46:58.243Z\"\r\n },\r\n {\r\n \"Id\": \"edaf1fb3-e787-4184-946f-cc0c378f156e\",\r\n \"Name\": \"user5@test.com\",\r\n \"ConnectorKey\": \"uipath-zendesk-zendesk\",\r\n \"ConnectorName\": \"Zendesk\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-05-13T11:03:12.636Z\",\r\n \"Updated\": \"2026-06-19T07:46:56.843Z\"\r\n },\r\n {\r\n \"Id\": \"0b78b83b-795d-49a2-951a-19306249f26b\",\r\n \"Name\": \"dev225223\",\r\n \"ConnectorKey\": \"uipath-servicenow-servicenow\",\r\n \"ConnectorName\": \"ServiceNow\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-05-13T11:07:09.220Z\",\r\n \"Updated\": \"2026-06-19T07:46:57.800Z\"\r\n },\r\n {\r\n \"Id\": \"1ff8b174-8f76-4249-a467-7846f10dd6b7\",\r\n \"Name\": \"user6@test.com-uipath-sandbox-380\",\r\n \"ConnectorKey\": \"uipath-atlassian-jira\",\r\n \"ConnectorName\": \"Jira\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-05-13T11:08:27.966Z\",\r\n \"Updated\": \"2026-06-19T07:46:57.516Z\"\r\n },\r\n {\r\n \"Id\": \"9d95be15-8820-466e-9d87-745f23e56d9f\",\r\n \"Name\": \"2144368\",\r\n \"ConnectorKey\": \"uipath-oracle-netsuite\",\r\n \"ConnectorName\": \"Oracle NetSuite\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-06-15T11:26:33.420Z\",\r\n \"Updated\": \"2026-06-19T07:46:57.283Z\"\r\n },\r\n {\r\n \"Id\": \"786ad982-8263-4ba2-941e-38dae2f5c8a8\",\r\n \"Name\": \"is-sandboxes\",\r\n \"ConnectorKey\": \"uipath-salesforce-slack\",\r\n \"ConnectorName\": \"Slack\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-06-17T07:01:22.243Z\",\r\n \"Updated\": \"2026-06-19T07:46:57.783Z\"\r\n },\r\n {\r\n \"Id\": \"ec84c4af-b354-4209-b388-5fbdb98ee800\",\r\n \"Name\": \"user5@test.com\",\r\n \"ConnectorKey\": \"uipath-citrix-sharefile\",\r\n \"ConnectorName\": \"Citrix ShareFile\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-06-17T07:08:04.443Z\",\r\n \"Updated\": \"2026-06-19T07:46:57.853Z\"\r\n },\r\n {\r\n \"Id\": \"65a6d5e5-7250-4a98-a7b9-931965b5ce4e\",\r\n \"Name\": \"uipath\",\r\n \"ConnectorKey\": \"uipath-freshworks-freshdesk\",\r\n \"ConnectorName\": \"Freshdesk\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"replacement_email@test.com\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"replacement_email@test.com's workspace\",\r\n \"FolderKey\": \"acdfea6d-632a-4327-94b3-e050717448d8\",\r\n \"Created\": \"2026-06-18T11:35:25.686Z\",\r\n \"Updated\": \"2026-06-19T07:46:57.366Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/task.yaml index c06b175945..c6ec5d1508 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-missing-dap/task.yaml @@ -15,21 +15,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/README.md new file mode 100644 index 0000000000..00de209a83 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/README.md @@ -0,0 +1,29 @@ +# rpa-is-connection-not-authorized-cns + +Reproduces the #3 Connection Service error by production volume: a connector +activity faults because the Integration Service connection is **no longer in +an authorized state** — the provider rejected token refresh with +`invalid_grant` — surfaced as `CNS1008` (HTTP 400) inside the activity's +`DAP-GE-3000` wrapper. The connection shows `State: Failed` and `ping` fails +with the same code. + +Fixtures are hand-authored from the production error signature (Connection +Service traces `StatusCode: BadRequest, ErrorCode: "CNS1008"`); shapes mirror +the sibling `rpa-is-*-dap` scenarios. + +## How this test reproduces it + +| Layer | Simulation | +|---|---| +| Faulted job | `or jobs list/get/logs`: `LeadImport` unattended run faulted at "Create Contact" with the `CNS1008` message embedding the provider's `invalid_grant` | +| Connection state | `is connections list` shows the Salesforce connection in **Failed** state; `ping` fails with HTTP 400 / `CNS1008` and re-authenticate instructions | +| Temporal signature | "Worked for weeks, broke this morning, nothing changed" — the token-expiry/revocation pattern | + +## Success criteria + +- `skill_triggered` — the uipath-troubleshoot skill actually ran +- `llm_judge` vs [RESOLUTION.md](./RESOLUTION.md) — root cause is the expired/revoked OAuth grant; fix is re-authenticating the existing connection (not recreating it, not a permissions change) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/02e8acc99f.json deleted file mode 100644 index 20f48d0af3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/1246fa0b59.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/1246fa0b59.json deleted file mode 100644 index 73d09fe0ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/1246fa0b59.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7", - "State": "Faulted", - "ProcessName": "LeadImport", - "Source": "Trigger", - "Type": "Unattended", - "Priority": "Normal", - "StartTime": "2026-07-08T05:30:04.220Z", - "EndTime": "2026-07-08T05:30:11.610Z", - "Id": 50263417 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/2e37ebc77e.json deleted file mode 100644 index 9d44d4483d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "CRM", - "ID": 2982110, - "Key": "2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6", - "Path": "Shared/CRM", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Marketing", - "ID": 2982004, - "Key": "8f0a1b2c-3d4e-4f50-a1b2-c3d4e5f6a7b8", - "Path": "Shared/Marketing", - "Type": "Standard", - "Description": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/6d68dea318.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/6d68dea318.json deleted file mode 100644 index 279db624e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/6d68dea318.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4", - "Name": "Salesforce - CRM", - "ConnectorKey": "uipath-salesforce-sfdc", - "ConnectorName": "Salesforce", - "State": "Failed", - "Owner": "crm.admin@acmecorp.test", - "IsDefault": "Yes", - "Folder": "CRM", - "FolderKey": "2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6", - "Created": "2026-05-12T09:14:02.950Z", - "Updated": "2026-07-06T07:50:51.870Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/README.md deleted file mode 100644 index 4c57a800f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/README.md +++ /dev/null @@ -1,25 +0,0 @@ -# rpa-is-connection-not-authorized-cns - -Reproduces the #3 Connection Service error by production volume: a connector -activity faults because the Integration Service connection is **no longer in -an authorized state** — the provider rejected token refresh with -`invalid_grant` — surfaced as `CNS1008` (HTTP 400) inside the activity's -`DAP-GE-3000` wrapper. The connection shows `State: Failed` and `ping` fails -with the same code. - -Fixtures are hand-authored from the production error signature (Connection -Service traces `StatusCode: BadRequest, ErrorCode: "CNS1008"`); shapes mirror -the sibling `rpa-is-*-dap` scenarios. - -## How this test reproduces it - -| Layer | Simulation | -|---|---| -| Faulted job | `or jobs list/get/logs`: `LeadImport` unattended run faulted at "Create Contact" with the `CNS1008` message embedding the provider's `invalid_grant` | -| Connection state | `is connections list` shows the Salesforce connection in **Failed** state; `ping` fails with HTTP 400 / `CNS1008` and re-authenticate instructions | -| Temporal signature | "Worked for weeks, broke this morning, nothing changed" — the token-expiry/revocation pattern | - -## Success criteria - -- `skill_triggered` — the uipath-troubleshoot skill actually ran -- `llm_judge` vs [RESOLUTION.md](./RESOLUTION.md) — root cause is the expired/revoked OAuth grant; fix is re-authenticating the existing connection (not recreating it, not a permissions change) diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b231d0bd69.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b231d0bd69.json deleted file mode 100644 index c41fd26f7d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b231d0bd69.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "HTTP 400: Connection [5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4] is not in an authorized state (error code CNS1008). Re-authenticate the connection to continue.", - "Instructions": "Run 'uip is connections edit ' to re-authenticate the connection." -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b4d2077dd5.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b4d2077dd5.json deleted file mode 100644 index 279db624e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b4d2077dd5.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4", - "Name": "Salesforce - CRM", - "ConnectorKey": "uipath-salesforce-sfdc", - "ConnectorName": "Salesforce", - "State": "Failed", - "Owner": "crm.admin@acmecorp.test", - "IsDefault": "Yes", - "Folder": "CRM", - "FolderKey": "2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6", - "Created": "2026-05-12T09:14:02.950Z", - "Updated": "2026-07-06T07:50:51.870Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b993a64dbd.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b993a64dbd.json deleted file mode 100644 index 26d506ad64..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/b993a64dbd.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7", - "StartTime": "2026-07-08T05:30:04.220Z", - "EndTime": "2026-07-08T05:30:11.610Z", - "State": "Faulted", - "JobPriority": "Normal", - "ResourceOverwrites": "{\"Connection.5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\": {\"EntityDisplayName\": \"crm.admin@acmecorp.test\", \"Level\": 0, \"Properties\": {\"connectionId\": \"5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\", \"folderKey\": \"2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6\", \"ConnectionId\": \"5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\"}}}", - "Source": "Trigger", - "SourceType": "Schedule", - "Info": "Job faulted at activity 'Create Contact': failed to retrieve connection (DAP-GE-3000)." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/caf0aa408c.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/caf0aa408c.json deleted file mode 100644 index c9645bf154..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/caf0aa408c.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-08T05:30:11.542Z", - "Message": "Create Contact: Failed to retrieve connection. Consider using a different connection. - Connection [5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4] is not in an authorized state. Please re-authenticate the connection to continue. Error code: DAP-GE-3000.\nin Main.xaml\n at ConnectorActivity \"Create Contact\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken cancellationToken)\n--- inner ---\nUiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection Service error reported: [StatusCode: BadRequest, ErrorCode: \"CNS1008\", ErrorReason: \"{\\\"error\\\":\\\"invalid_grant\\\",\\\"error_description\\\":\\\"expired access/refresh token\\\"}\", TraceId: \"3c5e7a9b1d2f46a8c0b2d4f6a8c0e1b3\"]", - "ProcessName": "LeadImport" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/manifest.json deleted file mode 100644 index 070399de7e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/m/r/manifest.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored for the CNS error-code playbooks (ENGCE-59477); fixtures reproduce the production CNS signature.", - "rules": [ - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key 2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6", - "file": "1246fa0b59.json" - }, - { - "match": "or jobs get 9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7", - "file": "b993a64dbd.json" - }, - { - "match": "or jobs logs 9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7", - "file": "caf0aa408c.json" - }, - { - "match": "is connections list --folder-key 2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6", - "file": "6d68dea318.json" - }, - { - "match": "is connections list --connection-id 5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4", - "file": "b4d2077dd5.json" - }, - { - "match": "is connections ping 5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4", - "file": "b231d0bd69.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/uip-fixture.json new file mode 100644 index 0000000000..6b387950d3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/data/uip-fixture.json @@ -0,0 +1,98 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"LeadImport\",\r\n \"Source\": \"Trigger\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"StartTime\": \"2026-07-08T05:30:04.220Z\",\r\n \"EndTime\": \"2026-07-08T05:30:11.610Z\",\r\n \"Id\": 50263417\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\",\r\n \"Name\": \"Salesforce - CRM\",\r\n \"ConnectorKey\": \"uipath-salesforce-sfdc\",\r\n \"ConnectorName\": \"Salesforce\",\r\n \"State\": \"Failed\",\r\n \"Owner\": \"crm.admin@acmecorp.test\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"CRM\",\r\n \"FolderKey\": \"2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6\",\r\n \"Created\": \"2026-05-12T09:14:02.950Z\",\r\n \"Updated\": \"2026-07-06T07:50:51.870Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--connection-id", + "5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\",\r\n \"Name\": \"Salesforce - CRM\",\r\n \"ConnectorKey\": \"uipath-salesforce-sfdc\",\r\n \"ConnectorName\": \"Salesforce\",\r\n \"State\": \"Failed\",\r\n \"Owner\": \"crm.admin@acmecorp.test\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"CRM\",\r\n \"FolderKey\": \"2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6\",\r\n \"Created\": \"2026-05-12T09:14:02.950Z\",\r\n \"Updated\": \"2026-07-06T07:50:51.870Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7\",\r\n \"StartTime\": \"2026-07-08T05:30:04.220Z\",\r\n \"EndTime\": \"2026-07-08T05:30:11.610Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"ResourceOverwrites\": \"{\\\"Connection.5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\\\": {\\\"EntityDisplayName\\\": \\\"crm.admin@acmecorp.test\\\", \\\"Level\\\": 0, \\\"Properties\\\": {\\\"connectionId\\\": \\\"5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\\\", \\\"folderKey\\\": \\\"2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6\\\", \\\"ConnectionId\\\": \\\"5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4\\\"}}}\",\r\n \"Source\": \"Trigger\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Job faulted at activity 'Create Contact': failed to retrieve connection (DAP-GE-3000).\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "9c1d3e5f-7a8b-4c2d-90e1-f2a3b4c5d6e7" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-08T05:30:11.542Z\",\r\n \"Message\": \"Create Contact: Failed to retrieve connection. Consider using a different connection. - Connection [5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4] is not in an authorized state. Please re-authenticate the connection to continue. Error code: DAP-GE-3000.\\nin Main.xaml\\n at ConnectorActivity \\\"Create Contact\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken cancellationToken)\\n--- inner ---\\nUiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection Service error reported: [StatusCode: BadRequest, ErrorCode: \\\"CNS1008\\\", ErrorReason: \\\"{\\\\\\\"error\\\\\\\":\\\\\\\"invalid_grant\\\\\\\",\\\\\\\"error_description\\\\\\\":\\\\\\\"expired access/refresh token\\\\\\\"}\\\", TraceId: \\\"3c5e7a9b1d2f46a8c0b2d4f6a8c0e1b3\\\"]\",\r\n \"ProcessName\": \"LeadImport\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"HTTP 400: Connection [5a7b9c1d-2e4f-4608-a1b3-c5d7e9f0a2b4] is not in an authorized state (error code CNS1008). Re-authenticate the connection to continue.\",\r\n \"Instructions\": \"Run 'uip is connections edit ' to re-authenticate the connection.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"CRM\",\r\n \"ID\": 2982110,\r\n \"Key\": \"2e4c6a80-1b3d-4f5e-8a90-c1d2e3f4a5b6\",\r\n \"Path\": \"Shared/CRM\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Marketing\",\r\n \"ID\": 2982004,\r\n \"Key\": \"8f0a1b2c-3d4e-4f50-a1b2-c3d4e5f6a7b8\",\r\n \"Path\": \"Shared/Marketing\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/task.yaml index d8832ef40b..aa63f79a04 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-connection-not-authorized-cns/task.yaml @@ -16,21 +16,12 @@ run_limits: turn_timeout: 1800 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/README.md new file mode 100644 index 0000000000..e18e515f6f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/README.md @@ -0,0 +1,31 @@ +# rpa-is-folder-permission-cns + +Reproduces the highest-volume *distinct-fix* Connection Service failure seen in +production telemetry: an unattended job faults at a connector activity because +the **robot account lacks the `Connections.View` folder permission** on the +folder holding the Integration Service connection — surfaced as `CNS1045` +inside the activity's `DAP-GE-3000` wrapper. The classic signature is +"debug works, unattended fails", which routinely gets misdiagnosed as a +connection-authentication problem. + +Fixtures are hand-authored from the production error signature (Connection +Service `FailureUserMessageKey: FolderAuth.403.CNS1045`) rather than extracted +from a live session; shapes mirror the sibling `rpa-is-*-dap` scenarios. + +## How this test reproduces it + +| Layer | Simulation | +|---|---| +| Faulted job | `or jobs list/get/logs` fixtures: `InvoiceSync` unattended run faulted at "Send Invoice Email" with the `CNS1045` message naming the permission and folder key | +| Connection state | `is connections ping` returns **Success** (the connection itself is healthy — the decisive evidence that re-authentication is the wrong fix) | +| Connection location | `is connections list --folder-key ` shows the Enabled Outlook connection; the job folder itself has none | +| Red herring | The "works in Studio" report tempts an auth/token diagnosis; the log's explicit permission message must win | + +## Success criteria + +- `skill_triggered` — the uipath-troubleshoot skill actually ran +- `llm_judge` vs [RESOLUTION.md](./RESOLUTION.md) — root cause is the robot's missing folder permission; fix is granting `Connections.View` (or moving the connection), explicitly **not** re-authentication + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/02e8acc99f.json deleted file mode 100644 index 20f48d0af3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/0baae564a8.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/0baae564a8.json deleted file mode 100644 index 1c63142804..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/0baae564a8.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5", - "State": "Faulted", - "ProcessName": "InvoiceSync", - "Source": "Trigger", - "Type": "Unattended", - "Priority": "Normal", - "StartTime": "2026-07-07T06:00:12.170Z", - "EndTime": "2026-07-07T06:00:19.440Z", - "Id": 50261001 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/2e37ebc77e.json deleted file mode 100644 index e29a5e7465..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "Finance", - "ID": 2980990, - "Key": "7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f", - "Path": "Shared/Finance", - "Type": "Standard", - "Description": "" - }, - { - "Name": "InvoiceOps", - "ID": 2981001, - "Key": "f1a2b3c4-9d0e-4f61-8a72-3b4c5d6e7f80", - "Path": "Shared/Finance/InvoiceOps", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Sales", - "ID": 2980721, - "Key": "0a1b2c3d-4e5f-4601-a2b3-c4d5e6f7a8b9", - "Path": "Shared/Sales", - "Type": "Standard", - "Description": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/49aa96c747.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/49aa96c747.json deleted file mode 100644 index 456e2d147d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/49aa96c747.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionPing", - "Data": { - "Id": "8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6", - "State": "Enabled", - "Message": "Connection is active and ready for operations." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/4b62941f74.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/4b62941f74.json deleted file mode 100644 index ddaecd5049..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/4b62941f74.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5", - "StartTime": "2026-07-07T06:00:12.170Z", - "EndTime": "2026-07-07T06:00:19.440Z", - "State": "Faulted", - "JobPriority": "Normal", - "ResourceOverwrites": "{\"Connection.8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\": {\"EntityDisplayName\": \"finance.ops@acmecorp.test\", \"Level\": 0, \"Properties\": {\"connectionId\": \"8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\", \"folderKey\": \"7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f\", \"ConnectionId\": \"8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\"}}}", - "Source": "Trigger", - "SourceType": "Schedule", - "Info": "Job faulted at activity 'Send Invoice Email': failed to retrieve connection (DAP-GE-3000)." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/6ec16f49db.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/6ec16f49db.json deleted file mode 100644 index 5e4e145bac..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/6ec16f49db.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/README.md deleted file mode 100644 index 595ad617a7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# rpa-is-folder-permission-cns - -Reproduces the highest-volume *distinct-fix* Connection Service failure seen in -production telemetry: an unattended job faults at a connector activity because -the **robot account lacks the `Connections.View` folder permission** on the -folder holding the Integration Service connection — surfaced as `CNS1045` -inside the activity's `DAP-GE-3000` wrapper. The classic signature is -"debug works, unattended fails", which routinely gets misdiagnosed as a -connection-authentication problem. - -Fixtures are hand-authored from the production error signature (Connection -Service `FailureUserMessageKey: FolderAuth.403.CNS1045`) rather than extracted -from a live session; shapes mirror the sibling `rpa-is-*-dap` scenarios. - -## How this test reproduces it - -| Layer | Simulation | -|---|---| -| Faulted job | `or jobs list/get/logs` fixtures: `InvoiceSync` unattended run faulted at "Send Invoice Email" with the `CNS1045` message naming the permission and folder key | -| Connection state | `is connections ping` returns **Success** (the connection itself is healthy — the decisive evidence that re-authentication is the wrong fix) | -| Connection location | `is connections list --folder-key ` shows the Enabled Outlook connection; the job folder itself has none | -| Red herring | The "works in Studio" report tempts an auth/token diagnosis; the log's explicit permission message must win | - -## Success criteria - -- `skill_triggered` — the uipath-troubleshoot skill actually ran -- `llm_judge` vs [RESOLUTION.md](./RESOLUTION.md) — root cause is the robot's missing folder permission; fix is granting `Connections.View` (or moving the connection), explicitly **not** re-authentication diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/a80d7a5467.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/a80d7a5467.json deleted file mode 100644 index ca78c9c9ab..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/a80d7a5467.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6", - "Name": "Outlook - Finance Ops", - "ConnectorKey": "uipath-microsoft-outlook365", - "ConnectorName": "Microsoft Outlook 365", - "State": "Enabled", - "Owner": "finance.ops@acmecorp.test", - "IsDefault": "Yes", - "Folder": "Finance", - "FolderKey": "7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f", - "Created": "2026-05-12T09:14:02.950Z", - "Updated": "2026-07-06T07:50:51.870Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/cec8826987.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/cec8826987.json deleted file mode 100644 index 686b62019d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/cec8826987.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-07T06:00:19.377Z", - "Message": "Send Invoice Email: Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: 7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f) Error code: DAP-GE-3000.\nin Main.xaml\n at ConnectorActivity \"Send Invoice Email\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken cancellationToken)\n--- inner ---\nUiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection Service error reported: [StatusCode: Forbidden, ErrorCode: \"CNS1045\", TraceId: \"7f3a9c1e5b2d48f6a0c4e8b2d6f0a3c5\"]", - "ProcessName": "InvoiceSync" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/manifest.json deleted file mode 100644 index ad32538b76..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/m/r/manifest.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored for the CNS error-code playbooks (ENGCE-59477); fixtures reproduce the production CNS signature.", - "rules": [ - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key f1a2b3c4-9d0e-4f61-8a72-3b4c5d6e7f80", - "file": "0baae564a8.json" - }, - { - "match": "or jobs get 4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5", - "file": "4b62941f74.json" - }, - { - "match": "or jobs logs 4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5", - "file": "cec8826987.json" - }, - { - "match": "is connections list --folder-key 7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f", - "file": "a80d7a5467.json" - }, - { - "match": "is connections list --folder-key f1a2b3c4-9d0e-4f61-8a72-3b4c5d6e7f80", - "file": "6ec16f49db.json" - }, - { - "match": "is connections ping 8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6", - "file": "49aa96c747.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/uip-fixture.json new file mode 100644 index 0000000000..a39bc069e8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/data/uip-fixture.json @@ -0,0 +1,98 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1a2b3c4-9d0e-4f61-8a72-3b4c5d6e7f80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceSync\",\r\n \"Source\": \"Trigger\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"StartTime\": \"2026-07-07T06:00:12.170Z\",\r\n \"EndTime\": \"2026-07-07T06:00:19.440Z\",\r\n \"Id\": 50261001\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\",\r\n \"Name\": \"Outlook - Finance Ops\",\r\n \"ConnectorKey\": \"uipath-microsoft-outlook365\",\r\n \"ConnectorName\": \"Microsoft Outlook 365\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"finance.ops@acmecorp.test\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"Finance\",\r\n \"FolderKey\": \"7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f\",\r\n \"Created\": \"2026-05-12T09:14:02.950Z\",\r\n \"Updated\": \"2026-07-06T07:50:51.870Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "f1a2b3c4-9d0e-4f61-8a72-3b4c5d6e7f80" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5\",\r\n \"StartTime\": \"2026-07-07T06:00:12.170Z\",\r\n \"EndTime\": \"2026-07-07T06:00:19.440Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"ResourceOverwrites\": \"{\\\"Connection.8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\\\": {\\\"EntityDisplayName\\\": \\\"finance.ops@acmecorp.test\\\", \\\"Level\\\": 0, \\\"Properties\\\": {\\\"connectionId\\\": \\\"8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\\\", \\\"folderKey\\\": \\\"7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f\\\", \\\"ConnectionId\\\": \\\"8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\\\"}}}\",\r\n \"Source\": \"Trigger\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Job faulted at activity 'Send Invoice Email': failed to retrieve connection (DAP-GE-3000).\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "4b8a2c6d-1e3f-4a50-b7c9-d0e1f2a3b4c5" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-07T06:00:19.377Z\",\r\n \"Message\": \"Send Invoice Email: Failed to retrieve connection. Consider using a different connection. - The robot does not have the Connections.View permission in the folder where this connection lives. Ask your administrator to grant Connections.View on that folder, or move the connection to a folder where the robot has the required permission. (Folder: 7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f) Error code: DAP-GE-3000.\\nin Main.xaml\\n at ConnectorActivity \\\"Send Invoice Email\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken cancellationToken)\\n--- inner ---\\nUiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection Service error reported: [StatusCode: Forbidden, ErrorCode: \\\"CNS1045\\\", TraceId: \\\"7f3a9c1e5b2d48f6a0c4e8b2d6f0a3c5\\\"]\",\r\n \"ProcessName\": \"InvoiceSync\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionPing\",\r\n \"Data\": {\r\n \"Id\": \"8d2f4a6b-3c5e-4710-92a4-b6c8d0e2f4a6\",\r\n \"State\": \"Enabled\",\r\n \"Message\": \"Connection is active and ready for operations.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"Finance\",\r\n \"ID\": 2980990,\r\n \"Key\": \"7c9e1f22-5b3a-4d48-9e10-6a2b8c4d0e5f\",\r\n \"Path\": \"Shared/Finance\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"InvoiceOps\",\r\n \"ID\": 2981001,\r\n \"Key\": \"f1a2b3c4-9d0e-4f61-8a72-3b4c5d6e7f80\",\r\n \"Path\": \"Shared/Finance/InvoiceOps\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Sales\",\r\n \"ID\": 2980721,\r\n \"Key\": \"0a1b2c3d-4e5f-4601-a2b3-c4d5e6f7a8b9\",\r\n \"Path\": \"Shared/Sales\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/task.yaml index 6281454a93..c4e337a261 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-folder-permission-cns/task.yaml @@ -16,21 +16,12 @@ run_limits: turn_timeout: 1800 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/README.md new file mode 100644 index 0000000000..7a9c70dffb --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/README.md @@ -0,0 +1,32 @@ +# rpa-is-personal-workspace-connection-cns + +Reproduces the classic cross-workspace Connection Service failure: a process +was published bound to a connection living in the **author's personal +workspace**, so the unattended robot cannot resolve it — Connection Service +returns **HTTP 404, error code `CNS1049`** ("connection is on a personal +folder"), surfaced as `DAP-GE-3000` at the connector activity. The trap this +scenario tests: a 404 here does NOT mean the connection was deleted, and +neither re-authentication nor recreating-in-place fixes it — the connection +must live in a shared folder the robot can reach. + +Fixtures are hand-authored from the production error signature (Connection +Service traces `StatusCode: NotFound, ErrorCode: "CNS1049"` with the +personal-folder message); shapes mirror the sibling `rpa-is-*-dap` scenarios. + +## How this test reproduces it + +| Layer | Simulation | +|---|---| +| Faulted job | `or jobs list/get/logs`: `NewHireDocs` unattended run faulted at "Upload Offer Letter" with the `CNS1049` personal-folder message | +| Job folder | `is connections list --folder-key ` returns **no connections** — nothing for the robot to resolve | +| The real location | `is connections list --connection-id ` shows the OneDrive connection alive and Enabled in "Personal Workspace - priya.sharma" | +| Disambiguation | `ping` fails 404 with the personal-folder message — deleted vs cross-workspace must be decided from the connection's folder, not the 404 alone | + +## Success criteria + +- `skill_triggered` — the uipath-troubleshoot skill actually ran +- `llm_judge` vs [RESOLUTION.md](./RESOLUTION.md) — root cause is the personal-workspace-bound connection; fix is creating/moving the connection into the shared folder and rebinding — **not** re-authentication and **not** "the connection was deleted" + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/02e8acc99f.json deleted file mode 100644 index 20f48d0af3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/1ed213503f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/1ed213503f.json deleted file mode 100644 index 5e4e145bac..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/1ed213503f.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/2e37ebc77e.json deleted file mode 100644 index 51c5be0dd7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "HR", - "ID": 2983200, - "Key": "3a4b5c6d-7e8f-4901-a2b3-c4d5e6f7a8b9", - "Path": "Shared/HR", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Onboarding", - "ID": 2983207, - "Key": "6b8d0f21-3a4c-4e5f-9012-a3b4c5d6e7f8", - "Path": "Shared/HR/Onboarding", - "Type": "Standard", - "Description": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/49a5cc2a82.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/49a5cc2a82.json deleted file mode 100644 index 2a24d5abe0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/49a5cc2a82.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "HTTP 404: Connection 'b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6' is on a personal folder and can only be used by its owner (error code CNS1049).", - "Instructions": "Create the connection in a shared folder the robot can access, then rebind the process to it." -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/62016f284b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/62016f284b.json deleted file mode 100644 index 026bd3885a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/62016f284b.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6", - "StartTime": "2026-07-08T04:15:02.330Z", - "EndTime": "2026-07-08T04:15:09.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "ResourceOverwrites": "{\"Connection.b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\": {\"EntityDisplayName\": \"priya.sharma@acmecorp.test\", \"Level\": 0, \"Properties\": {\"connectionId\": \"b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\", \"folderKey\": \"9e0f1a2b-3c4d-4e5f-a6b7-c8d9e0f1a2b3\", \"ConnectionId\": \"b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\"}}}", - "Source": "Trigger", - "SourceType": "Schedule", - "Info": "Job faulted at activity 'Upload Offer Letter': failed to retrieve connection (DAP-GE-3000)." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/README.md deleted file mode 100644 index bc9c5d0a2b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# rpa-is-personal-workspace-connection-cns - -Reproduces the classic cross-workspace Connection Service failure: a process -was published bound to a connection living in the **author's personal -workspace**, so the unattended robot cannot resolve it — Connection Service -returns **HTTP 404, error code `CNS1049`** ("connection is on a personal -folder"), surfaced as `DAP-GE-3000` at the connector activity. The trap this -scenario tests: a 404 here does NOT mean the connection was deleted, and -neither re-authentication nor recreating-in-place fixes it — the connection -must live in a shared folder the robot can reach. - -Fixtures are hand-authored from the production error signature (Connection -Service traces `StatusCode: NotFound, ErrorCode: "CNS1049"` with the -personal-folder message); shapes mirror the sibling `rpa-is-*-dap` scenarios. - -## How this test reproduces it - -| Layer | Simulation | -|---|---| -| Faulted job | `or jobs list/get/logs`: `NewHireDocs` unattended run faulted at "Upload Offer Letter" with the `CNS1049` personal-folder message | -| Job folder | `is connections list --folder-key ` returns **no connections** — nothing for the robot to resolve | -| The real location | `is connections list --connection-id ` shows the OneDrive connection alive and Enabled in "Personal Workspace - priya.sharma" | -| Disambiguation | `ping` fails 404 with the personal-folder message — deleted vs cross-workspace must be decided from the connection's folder, not the 404 alone | - -## Success criteria - -- `skill_triggered` — the uipath-troubleshoot skill actually ran -- `llm_judge` vs [RESOLUTION.md](./RESOLUTION.md) — root cause is the personal-workspace-bound connection; fix is creating/moving the connection into the shared folder and rebinding — **not** re-authentication and **not** "the connection was deleted" diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/aff14b30cf.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/aff14b30cf.json deleted file mode 100644 index 11b6c37ad7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/aff14b30cf.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6", - "Name": "priya.sharma@acmecorp.test", - "ConnectorKey": "uipath-microsoft-onedrive", - "ConnectorName": "Microsoft OneDrive & SharePoint", - "State": "Enabled", - "Owner": "priya.sharma@acmecorp.test", - "IsDefault": "Yes", - "Folder": "Personal Workspace - priya.sharma", - "FolderKey": "9e0f1a2b-3c4d-4e5f-a6b7-c8d9e0f1a2b3", - "Created": "2026-05-12T09:14:02.950Z", - "Updated": "2026-07-06T07:50:51.870Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/e316b114b6.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/e316b114b6.json deleted file mode 100644 index ccd5fdc141..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/e316b114b6.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6", - "State": "Faulted", - "ProcessName": "NewHireDocs", - "Source": "Trigger", - "Type": "Unattended", - "Priority": "Normal", - "StartTime": "2026-07-08T04:15:02.330Z", - "EndTime": "2026-07-08T04:15:09.870Z", - "Id": 50264892 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/f744f55949.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/f744f55949.json deleted file mode 100644 index 26e401eb51..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/f744f55949.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-08T04:15:09.801Z", - "Message": "Upload Offer Letter: Failed to retrieve connection. Consider using a different connection. - Connection 'b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6' is on a personal folder and can only be used by its owner. Error code: DAP-GE-3000.\nin Main.xaml\n at ConnectorActivity \"Upload Offer Letter\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken cancellationToken)\n--- inner ---\nUiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection Service error reported: [StatusCode: NotFound, ErrorCode: \"CNS1049\", TraceId: \"5d7f9b1c3e0a42c6e8a0b2c4d6e8f0a1\"]", - "ProcessName": "NewHireDocs" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/manifest.json deleted file mode 100644 index 5593315c8f..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/m/r/manifest.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored for the CNS error-code playbooks (ENGCE-59477); fixtures reproduce the production CNS signature.", - "rules": [ - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key 6b8d0f21-3a4c-4e5f-9012-a3b4c5d6e7f8", - "file": "e316b114b6.json" - }, - { - "match": "or jobs get 1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6", - "file": "62016f284b.json" - }, - { - "match": "or jobs logs 1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6", - "file": "f744f55949.json" - }, - { - "match": "is connections list --folder-key 6b8d0f21-3a4c-4e5f-9012-a3b4c5d6e7f8", - "file": "1ed213503f.json" - }, - { - "match": "is connections list --connection-id b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6", - "file": "aff14b30cf.json" - }, - { - "match": "is connections ping b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6", - "file": "49a5cc2a82.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/uip-fixture.json new file mode 100644 index 0000000000..c220f713c1 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/data/uip-fixture.json @@ -0,0 +1,98 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "6b8d0f21-3a4c-4e5f-9012-a3b4c5d6e7f8" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NewHireDocs\",\r\n \"Source\": \"Trigger\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"StartTime\": \"2026-07-08T04:15:02.330Z\",\r\n \"EndTime\": \"2026-07-08T04:15:09.870Z\",\r\n \"Id\": 50264892\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--folder-key", + "6b8d0f21-3a4c-4e5f-9012-a3b4c5d6e7f8" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--connection-id", + "b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\",\r\n \"Name\": \"priya.sharma@acmecorp.test\",\r\n \"ConnectorKey\": \"uipath-microsoft-onedrive\",\r\n \"ConnectorName\": \"Microsoft OneDrive & SharePoint\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"priya.sharma@acmecorp.test\",\r\n \"IsDefault\": \"Yes\",\r\n \"Folder\": \"Personal Workspace - priya.sharma\",\r\n \"FolderKey\": \"9e0f1a2b-3c4d-4e5f-a6b7-c8d9e0f1a2b3\",\r\n \"Created\": \"2026-05-12T09:14:02.950Z\",\r\n \"Updated\": \"2026-07-06T07:50:51.870Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6\",\r\n \"StartTime\": \"2026-07-08T04:15:02.330Z\",\r\n \"EndTime\": \"2026-07-08T04:15:09.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"ResourceOverwrites\": \"{\\\"Connection.b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\\\": {\\\"EntityDisplayName\\\": \\\"priya.sharma@acmecorp.test\\\", \\\"Level\\\": 0, \\\"Properties\\\": {\\\"connectionId\\\": \\\"b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\\\", \\\"folderKey\\\": \\\"9e0f1a2b-3c4d-4e5f-a6b7-c8d9e0f1a2b3\\\", \\\"ConnectionId\\\": \\\"b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6\\\"}}}\",\r\n \"Source\": \"Trigger\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Job faulted at activity 'Upload Offer Letter': failed to retrieve connection (DAP-GE-3000).\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "1f3a5c7e-9b0d-4e2f-8a4b-c6d8e0f2a4b6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-08T04:15:09.801Z\",\r\n \"Message\": \"Upload Offer Letter: Failed to retrieve connection. Consider using a different connection. - Connection 'b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6' is on a personal folder and can only be used by its owner. Error code: DAP-GE-3000.\\nin Main.xaml\\n at ConnectorActivity \\\"Upload Offer Letter\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.GeneralException\\n at UiPath.IntegrationService.Activities.Runtime.Services.ConnectionService.GetConnectionAsync(String connectionId, CancellationToken cancellationToken)\\n--- inner ---\\nUiPath.ConnectionClient.Contracts.ConnectionHttpException: Connection Service error reported: [StatusCode: NotFound, ErrorCode: \\\"CNS1049\\\", TraceId: \\\"5d7f9b1c3e0a42c6e8a0b2c4d6e8f0a1\\\"]\",\r\n \"ProcessName\": \"NewHireDocs\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"HTTP 404: Connection 'b4d6f8a0-c2e4-4671-8390-a1b2c3d4e5f6' is on a personal folder and can only be used by its owner (error code CNS1049).\",\r\n \"Instructions\": \"Create the connection in a shared folder the robot can access, then rebind the process to it.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"HR\",\r\n \"ID\": 2983200,\r\n \"Key\": \"3a4b5c6d-7e8f-4901-a2b3-c4d5e6f7a8b9\",\r\n \"Path\": \"Shared/HR\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Onboarding\",\r\n \"ID\": 2983207,\r\n \"Key\": \"6b8d0f21-3a4c-4e5f-9012-a3b4c5d6e7f8\",\r\n \"Path\": \"Shared/HR/Onboarding\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/task.yaml index 1199ab725e..c4a934908a 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-personal-workspace-connection-cns/task.yaml @@ -16,21 +16,12 @@ run_limits: turn_timeout: 1800 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/README.md new file mode 100644 index 0000000000..31d6096cbf --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/README.md @@ -0,0 +1,36 @@ +# Rpa Is Request Failed Dap — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +**Integration Service (root cause domain)** + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name rpa-is-request-failed-dap --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/02e8acc99f.json deleted file mode 100644 index 6205d04188..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/17b4f26aaf.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/17b4f26aaf.json deleted file mode 100644 index 13e15dfef4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/17b4f26aaf.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "29aae689-111d-459f-9a4a-d1ad1d9d2761", - "Name": "uipath", - "ConnectorKey": "uipath-freshworks-freshdesk", - "ConnectorName": "Freshdesk", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "No", - "Folder": "uipath-rpa-isActivities", - "FolderKey": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Created": "2026-06-18T13:24:02.950Z", - "Updated": "2026-06-18T14:26:54.873Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/2e37ebc77e.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/2e37ebc77e.json deleted file mode 100644 index b45a030686..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/2e37ebc77e.json +++ /dev/null @@ -1,406 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "e2e-folder-b-72c44bd7", - "ID": 2976049, - "Key": "fb61e5ba-40ef-4aca-9830-0a86dd6cfdae", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-72c44bd7", - "Type": "Standard", - "Description": "" - }, - { - "Name": "RequestFailed_DAPRuntime", - "ID": 2976014, - "Key": "e555fecd-20d3-47aa-864c-1c2e588d4769", - "Path": "Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime", - "Type": "Solution", - "Description": "" - }, - { - "Name": "uipath-rpa-isActivities", - "ID": 2975935, - "Key": "b262689d-e2a9-4c1a-8e68-04382a7fff76", - "Path": "Shared/uipath-rpa-isActivities", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-B", - "ID": 2975612, - "Key": "94f779c1-f602-49d1-a253-8a6bc18c0c45", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-A", - "ID": 2975609, - "Key": "fb57fea0-6855-4bd2-8b86-3bc9332a545f", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-94ece13e-top", - "ID": 2975605, - "Key": "7f121d04-2efb-4a6e-964a-2b00c7ac41e5", - "Path": "Shared/uipath-platform-e2e/e2e-fh-94ece13e-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-5b56c68d", - "ID": 2975488, - "Key": "3cbad2ee-ba8a-4d87-8048-985d7bb95336", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-5b56c68d", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-00fb23da", - "ID": 2975403, - "Key": "95d78e99-e937-417f-8e0c-82d6e46c04bd", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-00fb23da", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-22d1b6dd", - "ID": 2974296, - "Key": "2292cfbc-81ef-4252-a991-554de4bf2d48", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-22d1b6dd", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-acdd13ce", - "ID": 2974264, - "Key": "e1e0b094-425b-4a36-a7ec-0fe528e6ac1c", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-acdd13ce", - "Type": "Standard", - "Description": "" - }, - { - "Name": "CaseTest", - "ID": 2973309, - "Key": "583f75bc-3630-4d55-b269-5c8af1f17b80", - "Path": "Shared/uipath-maestro-case/CaseTest", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-deploy-folder-7188808a", - "ID": 2973247, - "Key": "40d377a0-0938-4e07-ad53-3e4b1f4dccdb", - "Path": "Shared/uipath-platform-e2e/e2e-deploy-folder-7188808a", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-folder-b-d6910c88", - "ID": 2973163, - "Key": "c1ddde01-465d-4684-a603-b2b96d87af94", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-d6910c88", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-0875838e", - "ID": 2973103, - "Key": "9cabacb6-8568-46e8-8cca-21a46f1a236b", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-0875838e", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-B", - "ID": 2973045, - "Key": "a909fdc9-4d95-48ab-9cf3-1f1c7e60d5de", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-A", - "ID": 2973044, - "Key": "c14bac5f-e6f0-4cc1-bb72-486f229421d2", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-b372c290-top", - "ID": 2973043, - "Key": "37fc9a90-9d49-414a-9158-c883eb0cd43b", - "Path": "Shared/uipath-platform-e2e/e2e-fh-b372c290-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-B", - "ID": 2955039, - "Key": "e66dbaa5-d7cf-4b9e-b66c-d42b372a2df1", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-A", - "ID": 2955037, - "Key": "46cc134a-e616-47ec-9e9a-360bf905ed5d", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-83886617-top", - "ID": 2955036, - "Key": "d6cd4cc4-a00e-4b87-8c46-4044385ddb3f", - "Path": "Shared/uipath-platform-e2e/e2e-fh-83886617-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-8df5d996", - "ID": 2954999, - "Key": "cd615438-aad8-4efe-af56-557bad7c1c2c", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-8df5d996", - "Type": "Standard", - "Description": "E2E test folder B for asset sharing" - }, - { - "Name": "e2e-folder-b-fce8a341", - "ID": 2954890, - "Key": "ad9bf36f-32e3-42b8-b051-c6cb114902ea", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-fce8a341", - "Type": "Standard", - "Description": "E2E test folder B for asset scoping isolation check" - }, - { - "Name": "e2e-fh-0b53372d-B", - "ID": 2952698, - "Key": "a1ef65e1-0213-4b50-ac52-e00b8235cad9", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-0b53372d-A", - "ID": 2952696, - "Key": "7a03267f-ec72-4cf6-a7a0-aa9128d8389a", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-0b53372d-top", - "ID": 2952694, - "Key": "c6f01161-4e94-4c59-a85a-76ac9b14963d", - "Path": "Shared/uipath-platform-e2e/e2e-fh-0b53372d-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-372553a0", - "ID": 2952623, - "Key": "5acebb10-f8d0-4c9a-a732-651f0f77a6ac", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-372553a0", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-e7972fc2", - "ID": 2952381, - "Key": "0347966b-d3db-4da4-b4ce-055d1b9a70a7", - "Path": "Shared/uipath-platform-e2e/e2e-folder-b-e7972fc2", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Solution 7", - "ID": 2933058, - "Key": "2c3f72f3-aaad-4cf6-b1bc-9e6504a3cb04", - "Path": "Shared/Solution 7", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-dr-debug", - "ID": 2923711, - "Key": "993f1502-37ff-43f4-81d9-3c6b040d8bbf", - "Path": "Shared/uipath-maestro-flow/Example-dr-debug", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-dailyreport", - "ID": 2923530, - "Key": "1c8e5662-0616-410f-b9e7-c01003b467c2", - "Path": "Shared/uipath-maestro-flow/Example-dailyreport", - "Type": "Solution", - "Description": "" - }, - { - "Name": "ExampleV3", - "ID": 2923525, - "Key": "1fc080e4-a221-44fe-a9b5-9f526637502f", - "Path": "Shared/ExampleV3", - "Type": "Solution", - "Description": "" - }, - { - "Name": "Example-debug", - "ID": 2923476, - "Key": "26315131-b943-4d72-9442-3476a0196bec", - "Path": "Shared/uipath-maestro-flow/Example-debug", - "Type": "Solution", - "Description": "" - }, - { - "Name": "FraudEscalation", - "ID": 2922399, - "Key": "8492394e-2f77-4b7d-96c1-c0b0662e3819", - "Path": "Shared/uipath-agents/FraudEscalation", - "Type": "Solution", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-B", - "ID": 2901931, - "Key": "fefa7912-d5d3-4dd5-87ea-3c4e3813321f", - "Path": "Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-B", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-A", - "ID": 2901928, - "Key": "44fd0e78-9eda-4a45-84f5-b2e07473f312", - "Path": "Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-A", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-fh-eecfa769-top", - "ID": 2901927, - "Key": "ffd6cb46-e6b8-4c58-94b4-3ebd480903c0", - "Path": "Shared/e2e-fh-eecfa769-top", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Finance", - "ID": 2901921, - "Key": "fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed", - "Path": "Finance", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-a-0a0a527c", - "ID": 2901912, - "Key": "c8c3f842-ef1f-4e9a-9728-191cba4bfd89", - "Path": "Shared/e2e-folder-a-0a0a527c", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-0a0a527c", - "ID": 2901910, - "Key": "0098b403-912d-48bf-983b-0bf7cc7a1240", - "Path": "Shared/e2e-folder-b-0a0a527c", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-a-8805f811", - "ID": 2901871, - "Key": "f573d08c-c8fa-485b-ad39-917b502eca45", - "Path": "Shared/e2e-folder-a-8805f811", - "Type": "Standard", - "Description": "E2E test folder A for asset sharing" - }, - { - "Name": "e2e-folder-b-8805f811", - "ID": 2901870, - "Key": "66247ea0-e4ad-4eb5-989d-846926c0acd5", - "Path": "Shared/e2e-folder-b-8805f811", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-a441329c", - "ID": 2901830, - "Key": "b5eb7093-74cf-443c-8d02-8712f4831c5a", - "Path": "Shared/e2e-folder-b-a441329c", - "Type": "Standard", - "Description": "E2E test folder B for asset sharing" - }, - { - "Name": "e2e-folder-a-a441329c", - "ID": 2901829, - "Key": "616562ea-0b92-41bc-9c3e-110c3981a647", - "Path": "Shared/e2e-folder-a-a441329c", - "Type": "Standard", - "Description": "E2E test folder A for asset sharing" - }, - { - "Name": "e2e-folder-a-4c752284", - "ID": 2901827, - "Key": "abd2c82d-ddc3-4c1d-8c9e-6da3a0321921", - "Path": "Shared/e2e-folder-a-4c752284", - "Type": "Standard", - "Description": "" - }, - { - "Name": "e2e-folder-b-4c752284", - "ID": 2901826, - "Key": "8a06458d-c42e-4f8b-9295-447bbdf2702e", - "Path": "Shared/e2e-folder-b-4c752284", - "Type": "Standard", - "Description": "" - }, - { - "Name": "FinancialPostingFunction", - "ID": 2897454, - "Key": "aa52840c-acc5-44f7-ac5b-9f67f6453ed1", - "Path": "Shared/uipath-maestro-flow/BillingDispute/FinancialPostingFunction", - "Type": "Solution", - "Description": "" - }, - { - "Name": "BillingDispute", - "ID": 2897440, - "Key": "79a7d12e-da84-4bf4-ab60-7f34b44dd178", - "Path": "Shared/uipath-maestro-flow/BillingDispute", - "Type": "Standard", - "Description": "" - }, - { - "Name": "acme-echo-solution 1", - "ID": 2895681, - "Key": "49f00a14-48be-405d-8568-976dbc6e1594", - "Path": "Shared/acme-echo-solution 1", - "Type": "Solution", - "Description": "" - }, - { - "Name": "acme-echo-solution", - "ID": 2895676, - "Key": "726471ba-7358-4d4e-ab6e-980698eccd1c", - "Path": "Shared/acme-echo-solution", - "Type": "Solution", - "Description": "" - }, - { - "Name": "uipath-platform-e2e", - "ID": 2892937, - "Key": "c23e5354-4c28-4a8d-ab8e-d63844b35926", - "Path": "Shared/uipath-platform-e2e", - "Type": "Standard", - "Description": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/3db6b96527.json deleted file mode 100644 index db9158e17c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/3db6b96527.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "38c2b236-c392-420f-9256-b50467e02122", - "StartTime": "2026-06-18T13:31:45.133Z", - "EndTime": "2026-06-18T13:31:50.893Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "ResourceOverwrites": "{\"Connection.29aae689-111d-459f-9a4a-d1ad1d9d2761\":{\"EntityDisplayName\":\"uipath\",\"Level\":0,\"Properties\":{\"connectionId\":\"29aae689-111d-459f-9a4a-d1ad1d9d2761\",\"folderKey\":\"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\"ConnectionId\":\"29aae689-111d-459f-9a4a-d1ad1d9d2761\"}}}", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d014c03c-ed7a-4bd8-b083-d4d59af37049", - "Info": "Request failed with error: \nProviderMessage : code - access_denied, message - You are not authorized to perform this action.\nProviderErrorCode : 403\nRequestId : 6a33f346e4b089c6407a42a2\nMessage : Forbidden\n\r\nStatus code: Forbidden. Error code: DAP-RT-1101.\r\n\r\nin Main.xaml\n at ConnectorActivity \"Create Ticket\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Request failed with error: \nProviderMessage : code - access_denied, message - You are not authorized to perform this action.\nProviderErrorCode : 403\nRequestId : 6a33f346e4b089c6407a42a2\nMessage : Forbidden\n\r\nStatus code: Forbidden. Error code: DAP-RT-1101.\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendCUDRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-18T13:31:42.187Z", - "ReleaseName": "RPA Workflow", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "BBCFBBA7-8063-4F00-E5D1-0B49C871596F", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Serverless", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1088623, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 2976014, - "FolderKey": "e555fecd-20d3-47aa-864c-1c2e588d4769", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": true, - "LocalSystemAccount": "ROBOTUSER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "ServerlessJobType": "RobotJob", - "LastModificationTime": "2026-06-18T13:31:50.893Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "38c2b236c392420f9256b50467e02122", - "Id": 50205597 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/701433db1c.json deleted file mode 100644 index 8c5a15300c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/701433db1c.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-18T13:31:50.900Z", - "JobId": 50205597 - }, - { - "State": "Running", - "Timestamp": "2026-06-18T13:31:45.143Z", - "JobId": 50205597 - }, - { - "State": "Pending", - "Timestamp": "2026-06-18T13:31:42.220Z", - "JobId": 50205597 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/7fbd91164b.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/7fbd91164b.json deleted file mode 100644 index 51c40f96a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/7fbd91164b.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobCompleted", - "Data": { - "Key": "fe6a1c07-c860-4608-a8f8-2a6ad08b494a", - "StartTime": "2026-06-18T14:51:29.653Z", - "EndTime": "2026-06-18T14:51:37.743Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "ResourceOverwrites": "{\"Connection.29aae689-111d-459f-9a4a-d1ad1d9d2761\":{\"EntityDisplayName\":\"uipath\",\"Level\":0,\"Properties\":{\"connectionId\":\"29aae689-111d-459f-9a4a-d1ad1d9d2761\",\"folderKey\":\"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\"ConnectionId\":\"29aae689-111d-459f-9a4a-d1ad1d9d2761\"}}}", - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "42f9f488-6f0f-4933-b445-0f762eb30eb4", - "Info": "Job completed", - "CreationTime": "2026-06-18T14:51:21.217Z", - "ReleaseName": "RPA Workflow", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "A5F32106-3865-4F00-3DC3-0B49C87C8612", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Serverless", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1088623, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 2976014, - "FolderKey": "e555fecd-20d3-47aa-864c-1c2e588d4769", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": true, - "LocalSystemAccount": "ROBOTUSER", - "OrchestratorUserIdentity": "default robot", - "RemoteControlAccess": "None", - "ServerlessJobType": "RobotJob", - "LastModificationTime": "2026-06-18T14:51:37.743Z", - "TraceId": "fe6a1c07c8604608a8f82a6ad08b494a", - "Id": 50208799, - "Duration": "17s" - } -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/88cdb52c69.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/88cdb52c69.json deleted file mode 100644 index ec3bd791f1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/88cdb52c69.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "ProcessList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "BC928F9D-34FB-45AB-8823-90654F2F4BB0", - "Name": "RPA Workflow", - "ProcessKey": "RequestFailed_DAPRuntime.process.RPA.Workflow", - "ProcessVersion": "1.0.0", - "Description": "", - "IsLatestVersion": true, - "RetentionPeriod": 30, - "RetentionAction": "Delete" - } - ] -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/9983dfffc0.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/9983dfffc0.json deleted file mode 100644 index b119a41e76..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/9983dfffc0.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionPing", - "Data": { - "ConnectionId": "29aae689-111d-459f-9a4a-d1ad1d9d2761", - "Status": "Enabled", - "Message": "Connection is active and ready for operations." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/README.md deleted file mode 100644 index c74a3c6bfe..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Rpa Is Request Failed Dap — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -**Integration Service (root cause domain)** - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name rpa-is-request-failed-dap --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b5848ea1e7.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b5848ea1e7.json deleted file mode 100644 index f753418bd4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b5848ea1e7.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "Result": "Success", - "Code": "ResourceMetadata", - "Data": { - "name": "tickets", - "displayName": "Tickets", - "availableOperations": [ - "List", - "Retrieve", - "Create", - "Replace", - "Delete" - ], - "files": { - "List": "~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.List.json", - "Retrieve": "~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Retrieve.json", - "Create": "~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Create.json", - "Replace": "~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Replace.json", - "Delete": "~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Delete.json" - }, - "hint": "Use --operation to see fields, or read the files directly for full details.", - "Warning": "Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields. Run 'is connections list ' to get available connection IDs." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b8d18266dc.json deleted file mode 100644 index d858edae8e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-18T13:31:50.711Z", - "Message": "RPA Workflow execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-18T13:31:50.662Z", - "Message": "Create Ticket: Request failed with error: \nProviderMessage : code - access_denied, message - You are not authorized to perform this action.\nProviderErrorCode : 403\nRequestId : 6a33f346e4b089c6407a42a2\nMessage : Forbidden\n\r\nStatus code: Forbidden. Error code: DAP-RT-1101.\nin Main.xaml\n at ConnectorActivity \"Create Ticket\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendCUDRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)" - }, - { - "Level": "Info", - "TimeStamp": "2026-06-18T13:31:47.931Z", - "Message": "RPA Workflow execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b9024389e9.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b9024389e9.json deleted file mode 100644 index c256534d24..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/b9024389e9.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "Message", - "Data": { - "Message": "No connections found for connection ID '29aae689-111d-459f-9a4a-d1ad1d9d2761'." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/bc42a35b46.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/bc42a35b46.json deleted file mode 100644 index fdaf83996a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/bc42a35b46.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "Result": "Success", - "Code": "ResourceMetadata", - "Data": { - "name": "tickets", - "displayName": "Create Ticket", - "path": "/tickets", - "operation": "Create", - "metadataFile": "~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Create.json", - "requiredFields": [], - "optionalFields": [ - "subject", - "source", - "type", - "product_id", - "twitter_id", - "email_config_id", - "priority", - "facebook_id", - "phone", - "parent_id", - "internal_group_id", - "name", - "unique_external_id", - "requester_id", - "status", - "responder_id", - "description", - "internal_agent_id", - "email", - "company_id", - "cc_emails[*]", - "tags[*]", - "group_id" - ], - "responseFields": [ - "requester.mobile", - "requester.name", - "subject", - "fr_escalated", - "source", - "type", - "product_id", - "id", - "twitter_id", - "stats.first_responded_at", - "email_config_id", - "description_text", - "priority", - "facebook_id", - "is_escalated", - "ticket_cc_emails[*]", - "phone", - "parent_id", - "internal_group_id", - "name", - "unique_external_id", - "requester_id", - "status", - "requester.email", - "responder_id", - "due_by", - "stats.agent_responded_at", - "description", - "created_at", - "reply_cc_emails[*]", - "requester.id", - "requester.phone", - "updated_at", - "internal_agent_id", - "email", - "company_id", - "form_id", - "fr_due_by", - "cc_emails[*]", - "tags[*]", - "stats.status_updated_at", - "nr_escalated", - "group_id", - "spam" - ], - "Warning": "Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields. Run 'is connections list ' to get available connection IDs." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/f002713ee7.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/f002713ee7.json deleted file mode 100644 index 5fb92f4600..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/f002713ee7.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "38c2b236-c392-420f-9256-b50467e02122", - "State": "Faulted", - "ProcessName": "RPA Workflow", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "BBCFBBA7-8063-4F00-E5D1-0B49C871596F", - "StartTime": "2026-06-18T13:31:45.133Z", - "EndTime": "2026-06-18T13:31:50.893Z", - "CreationTime": "2026-06-18T13:31:42.187Z", - "Id": 50205597 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/manifest.json deleted file mode 100644 index 973eb6726d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/m/r/manifest.json +++ /dev/null @@ -1,67 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or processes list --folder-path \"Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime\" --output json", - "file": "88cdb52c69.json" - }, - { - "match": "or jobs start BC928F9D-34FB-45AB-8823-90654F2F4BB0 --folder-path \"Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime\" --wait-for-completion --timeout 300 --output json", - "file": "7fbd91164b.json" - }, - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "is connections ping 29aae689-111d-459f-9a4a-d1ad1d9d2761", - "file": "9983dfffc0.json" - }, - { - "match": "is resources describe uipath-freshworks-freshdesk tickets --operation Create", - "file": "bc42a35b46.json" - }, - { - "match": "is resources describe uipath-freshworks-freshdesk tickets", - "file": "b5848ea1e7.json" - }, - { - "match": "is connections list --connection-id 29aae689-111d-459f-9a4a-d1ad1d9d2761", - "file": "b9024389e9.json" - }, - { - "match": "is connections list uipath-freshworks-freshdesk --folder-key b262689d-e2a9-4c1a-8e68-04382a7fff76", - "file": "17b4f26aaf.json" - }, - { - "match": "or folders list --output json", - "file": "2e37ebc77e.json" - }, - { - "match": "or jobs list --folder-key e555fecd-20d3-47aa-864c-1c2e588d4769 --state Faulted --output json", - "file": "f002713ee7.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/uip-fixture.json new file mode 100644 index 0000000000..3c61e5ca5a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/data/uip-fixture.json @@ -0,0 +1,169 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "start", + "BC928F9D-34FB-45AB-8823-90654F2F4BB0", + "--folder-path", + "Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime", + "--wait-for-completion", + "--timeout", + "300", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobCompleted\",\r\n \"Data\": {\r\n \"Key\": \"fe6a1c07-c860-4608-a8f8-2a6ad08b494a\",\r\n \"StartTime\": \"2026-06-18T14:51:29.653Z\",\r\n \"EndTime\": \"2026-06-18T14:51:37.743Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"ResourceOverwrites\": \"{\\\"Connection.29aae689-111d-459f-9a4a-d1ad1d9d2761\\\":{\\\"EntityDisplayName\\\":\\\"uipath\\\",\\\"Level\\\":0,\\\"Properties\\\":{\\\"connectionId\\\":\\\"29aae689-111d-459f-9a4a-d1ad1d9d2761\\\",\\\"folderKey\\\":\\\"b262689d-e2a9-4c1a-8e68-04382a7fff76\\\",\\\"ConnectionId\\\":\\\"29aae689-111d-459f-9a4a-d1ad1d9d2761\\\"}}}\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"42f9f488-6f0f-4933-b445-0f762eb30eb4\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-06-18T14:51:21.217Z\",\r\n \"ReleaseName\": \"RPA Workflow\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"A5F32106-3865-4F00-3DC3-0B49C87C8612\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Serverless\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1088623,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 2976014,\r\n \"FolderKey\": \"e555fecd-20d3-47aa-864c-1c2e588d4769\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": true,\r\n \"LocalSystemAccount\": \"ROBOTUSER\",\r\n \"OrchestratorUserIdentity\": \"default robot\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"ServerlessJobType\": \"RobotJob\",\r\n \"LastModificationTime\": \"2026-06-18T14:51:37.743Z\",\r\n \"TraceId\": \"fe6a1c07c8604608a8f82a6ad08b494a\",\r\n \"Id\": 50208799,\r\n \"Duration\": \"17s\"\r\n }\r\n}" + }, + { + "argv": [ + "is", + "resources", + "describe", + "uipath-freshworks-freshdesk", + "tickets", + "--operation", + "Create" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ResourceMetadata\",\r\n \"Data\": {\r\n \"name\": \"tickets\",\r\n \"displayName\": \"Create Ticket\",\r\n \"path\": \"/tickets\",\r\n \"operation\": \"Create\",\r\n \"metadataFile\": \"~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Create.json\",\r\n \"requiredFields\": [],\r\n \"optionalFields\": [\r\n \"subject\",\r\n \"source\",\r\n \"type\",\r\n \"product_id\",\r\n \"twitter_id\",\r\n \"email_config_id\",\r\n \"priority\",\r\n \"facebook_id\",\r\n \"phone\",\r\n \"parent_id\",\r\n \"internal_group_id\",\r\n \"name\",\r\n \"unique_external_id\",\r\n \"requester_id\",\r\n \"status\",\r\n \"responder_id\",\r\n \"description\",\r\n \"internal_agent_id\",\r\n \"email\",\r\n \"company_id\",\r\n \"cc_emails[*]\",\r\n \"tags[*]\",\r\n \"group_id\"\r\n ],\r\n \"responseFields\": [\r\n \"requester.mobile\",\r\n \"requester.name\",\r\n \"subject\",\r\n \"fr_escalated\",\r\n \"source\",\r\n \"type\",\r\n \"product_id\",\r\n \"id\",\r\n \"twitter_id\",\r\n \"stats.first_responded_at\",\r\n \"email_config_id\",\r\n \"description_text\",\r\n \"priority\",\r\n \"facebook_id\",\r\n \"is_escalated\",\r\n \"ticket_cc_emails[*]\",\r\n \"phone\",\r\n \"parent_id\",\r\n \"internal_group_id\",\r\n \"name\",\r\n \"unique_external_id\",\r\n \"requester_id\",\r\n \"status\",\r\n \"requester.email\",\r\n \"responder_id\",\r\n \"due_by\",\r\n \"stats.agent_responded_at\",\r\n \"description\",\r\n \"created_at\",\r\n \"reply_cc_emails[*]\",\r\n \"requester.id\",\r\n \"requester.phone\",\r\n \"updated_at\",\r\n \"internal_agent_id\",\r\n \"email\",\r\n \"company_id\",\r\n \"form_id\",\r\n \"fr_due_by\",\r\n \"cc_emails[*]\",\r\n \"tags[*]\",\r\n \"stats.status_updated_at\",\r\n \"nr_escalated\",\r\n \"group_id\",\r\n \"spam\"\r\n ],\r\n \"Warning\": \"Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields. Run 'is connections list ' to get available connection IDs.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e555fecd-20d3-47aa-864c-1c2e588d4769", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"38c2b236-c392-420f-9256-b50467e02122\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"RPA Workflow\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"BBCFBBA7-8063-4F00-E5D1-0B49C871596F\",\r\n \"StartTime\": \"2026-06-18T13:31:45.133Z\",\r\n \"EndTime\": \"2026-06-18T13:31:50.893Z\",\r\n \"CreationTime\": \"2026-06-18T13:31:42.187Z\",\r\n \"Id\": 50205597\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "uipath-freshworks-freshdesk", + "--folder-key", + "b262689d-e2a9-4c1a-8e68-04382a7fff76" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"29aae689-111d-459f-9a4a-d1ad1d9d2761\",\r\n \"Name\": \"uipath\",\r\n \"ConnectorKey\": \"uipath-freshworks-freshdesk\",\r\n \"ConnectorName\": \"Freshdesk\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"original_email@test.com\",\r\n \"IsDefault\": \"No\",\r\n \"Folder\": \"uipath-rpa-isActivities\",\r\n \"FolderKey\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Created\": \"2026-06-18T13:24:02.950Z\",\r\n \"Updated\": \"2026-06-18T14:26:54.873Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "processes", + "list", + "--folder-path", + "Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ProcessList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"BC928F9D-34FB-45AB-8823-90654F2F4BB0\",\r\n \"Name\": \"RPA Workflow\",\r\n \"ProcessKey\": \"RequestFailed_DAPRuntime.process.RPA.Workflow\",\r\n \"ProcessVersion\": \"1.0.0\",\r\n \"Description\": \"\",\r\n \"IsLatestVersion\": true,\r\n \"RetentionPeriod\": 30,\r\n \"RetentionAction\": \"Delete\"\r\n }\r\n ]\r\n}" + }, + { + "argv": [ + "is", + "resources", + "describe", + "uipath-freshworks-freshdesk", + "tickets" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ResourceMetadata\",\r\n \"Data\": {\r\n \"name\": \"tickets\",\r\n \"displayName\": \"Tickets\",\r\n \"availableOperations\": [\r\n \"List\",\r\n \"Retrieve\",\r\n \"Create\",\r\n \"Replace\",\r\n \"Delete\"\r\n ],\r\n \"files\": {\r\n \"List\": \"~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.List.json\",\r\n \"Retrieve\": \"~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Retrieve.json\",\r\n \"Create\": \"~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Create.json\",\r\n \"Replace\": \"~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Replace.json\",\r\n \"Delete\": \"~/.uipath/cache/integrationservice/uipath-freshworks-freshdesk/_static/tickets.Delete.json\"\r\n },\r\n \"hint\": \"Use --operation to see fields, or read the files directly for full details.\",\r\n \"Warning\": \"Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields. Run 'is connections list ' to get available connection IDs.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--connection-id", + "29aae689-111d-459f-9a4a-d1ad1d9d2761" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"Message\",\r\n \"Data\": {\r\n \"Message\": \"No connections found for connection ID '29aae689-111d-459f-9a4a-d1ad1d9d2761'.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping", + "29aae689-111d-459f-9a4a-d1ad1d9d2761" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionPing\",\r\n \"Data\": {\r\n \"ConnectionId\": \"29aae689-111d-459f-9a4a-d1ad1d9d2761\",\r\n \"Status\": \"Enabled\",\r\n \"Message\": \"Connection is active and ready for operations.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"e2e-folder-b-72c44bd7\",\r\n \"ID\": 2976049,\r\n \"Key\": \"fb61e5ba-40ef-4aca-9830-0a86dd6cfdae\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-72c44bd7\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"RequestFailed_DAPRuntime\",\r\n \"ID\": 2976014,\r\n \"Key\": \"e555fecd-20d3-47aa-864c-1c2e588d4769\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities/RequestFailed_DAPRuntime\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"uipath-rpa-isActivities\",\r\n \"ID\": 2975935,\r\n \"Key\": \"b262689d-e2a9-4c1a-8e68-04382a7fff76\",\r\n \"Path\": \"Shared/uipath-rpa-isActivities\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-B\",\r\n \"ID\": 2975612,\r\n \"Key\": \"94f779c1-f602-49d1-a253-8a6bc18c0c45\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-A\",\r\n \"ID\": 2975609,\r\n \"Key\": \"fb57fea0-6855-4bd2-8b86-3bc9332a545f\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top/e2e-fh-94ece13e-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-94ece13e-top\",\r\n \"ID\": 2975605,\r\n \"Key\": \"7f121d04-2efb-4a6e-964a-2b00c7ac41e5\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-94ece13e-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-5b56c68d\",\r\n \"ID\": 2975488,\r\n \"Key\": \"3cbad2ee-ba8a-4d87-8048-985d7bb95336\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-5b56c68d\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-00fb23da\",\r\n \"ID\": 2975403,\r\n \"Key\": \"95d78e99-e937-417f-8e0c-82d6e46c04bd\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-00fb23da\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-22d1b6dd\",\r\n \"ID\": 2974296,\r\n \"Key\": \"2292cfbc-81ef-4252-a991-554de4bf2d48\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-22d1b6dd\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-acdd13ce\",\r\n \"ID\": 2974264,\r\n \"Key\": \"e1e0b094-425b-4a36-a7ec-0fe528e6ac1c\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-acdd13ce\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"CaseTest\",\r\n \"ID\": 2973309,\r\n \"Key\": \"583f75bc-3630-4d55-b269-5c8af1f17b80\",\r\n \"Path\": \"Shared/uipath-maestro-case/CaseTest\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-deploy-folder-7188808a\",\r\n \"ID\": 2973247,\r\n \"Key\": \"40d377a0-0938-4e07-ad53-3e4b1f4dccdb\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-deploy-folder-7188808a\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-d6910c88\",\r\n \"ID\": 2973163,\r\n \"Key\": \"c1ddde01-465d-4684-a603-b2b96d87af94\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-d6910c88\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-0875838e\",\r\n \"ID\": 2973103,\r\n \"Key\": \"9cabacb6-8568-46e8-8cca-21a46f1a236b\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-0875838e\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-B\",\r\n \"ID\": 2973045,\r\n \"Key\": \"a909fdc9-4d95-48ab-9cf3-1f1c7e60d5de\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-A\",\r\n \"ID\": 2973044,\r\n \"Key\": \"c14bac5f-e6f0-4cc1-bb72-486f229421d2\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top/e2e-fh-b372c290-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-b372c290-top\",\r\n \"ID\": 2973043,\r\n \"Key\": \"37fc9a90-9d49-414a-9158-c883eb0cd43b\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-b372c290-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-B\",\r\n \"ID\": 2955039,\r\n \"Key\": \"e66dbaa5-d7cf-4b9e-b66c-d42b372a2df1\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-A\",\r\n \"ID\": 2955037,\r\n \"Key\": \"46cc134a-e616-47ec-9e9a-360bf905ed5d\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top/e2e-fh-83886617-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-83886617-top\",\r\n \"ID\": 2955036,\r\n \"Key\": \"d6cd4cc4-a00e-4b87-8c46-4044385ddb3f\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-83886617-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-8df5d996\",\r\n \"ID\": 2954999,\r\n \"Key\": \"cd615438-aad8-4efe-af56-557bad7c1c2c\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-8df5d996\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-fce8a341\",\r\n \"ID\": 2954890,\r\n \"Key\": \"ad9bf36f-32e3-42b8-b051-c6cb114902ea\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-fce8a341\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset scoping isolation check\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-B\",\r\n \"ID\": 2952698,\r\n \"Key\": \"a1ef65e1-0213-4b50-ac52-e00b8235cad9\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-A\",\r\n \"ID\": 2952696,\r\n \"Key\": \"7a03267f-ec72-4cf6-a7a0-aa9128d8389a\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top/e2e-fh-0b53372d-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-0b53372d-top\",\r\n \"ID\": 2952694,\r\n \"Key\": \"c6f01161-4e94-4c59-a85a-76ac9b14963d\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-fh-0b53372d-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-372553a0\",\r\n \"ID\": 2952623,\r\n \"Key\": \"5acebb10-f8d0-4c9a-a732-651f0f77a6ac\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-372553a0\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-e7972fc2\",\r\n \"ID\": 2952381,\r\n \"Key\": \"0347966b-d3db-4da4-b4ce-055d1b9a70a7\",\r\n \"Path\": \"Shared/uipath-platform-e2e/e2e-folder-b-e7972fc2\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Solution 7\",\r\n \"ID\": 2933058,\r\n \"Key\": \"2c3f72f3-aaad-4cf6-b1bc-9e6504a3cb04\",\r\n \"Path\": \"Shared/Solution 7\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-dr-debug\",\r\n \"ID\": 2923711,\r\n \"Key\": \"993f1502-37ff-43f4-81d9-3c6b040d8bbf\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-dr-debug\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-dailyreport\",\r\n \"ID\": 2923530,\r\n \"Key\": \"1c8e5662-0616-410f-b9e7-c01003b467c2\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-dailyreport\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"ExampleV3\",\r\n \"ID\": 2923525,\r\n \"Key\": \"1fc080e4-a221-44fe-a9b5-9f526637502f\",\r\n \"Path\": \"Shared/ExampleV3\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Example-debug\",\r\n \"ID\": 2923476,\r\n \"Key\": \"26315131-b943-4d72-9442-3476a0196bec\",\r\n \"Path\": \"Shared/uipath-maestro-flow/Example-debug\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"FraudEscalation\",\r\n \"ID\": 2922399,\r\n \"Key\": \"8492394e-2f77-4b7d-96c1-c0b0662e3819\",\r\n \"Path\": \"Shared/uipath-agents/FraudEscalation\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-B\",\r\n \"ID\": 2901931,\r\n \"Key\": \"fefa7912-d5d3-4dd5-87ea-3c4e3813321f\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-B\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-A\",\r\n \"ID\": 2901928,\r\n \"Key\": \"44fd0e78-9eda-4a45-84f5-b2e07473f312\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top/e2e-fh-eecfa769-A\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-fh-eecfa769-top\",\r\n \"ID\": 2901927,\r\n \"Key\": \"ffd6cb46-e6b8-4c58-94b4-3ebd480903c0\",\r\n \"Path\": \"Shared/e2e-fh-eecfa769-top\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Finance\",\r\n \"ID\": 2901921,\r\n \"Key\": \"fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed\",\r\n \"Path\": \"Finance\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-0a0a527c\",\r\n \"ID\": 2901912,\r\n \"Key\": \"c8c3f842-ef1f-4e9a-9728-191cba4bfd89\",\r\n \"Path\": \"Shared/e2e-folder-a-0a0a527c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-0a0a527c\",\r\n \"ID\": 2901910,\r\n \"Key\": \"0098b403-912d-48bf-983b-0bf7cc7a1240\",\r\n \"Path\": \"Shared/e2e-folder-b-0a0a527c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-8805f811\",\r\n \"ID\": 2901871,\r\n \"Key\": \"f573d08c-c8fa-485b-ad39-917b502eca45\",\r\n \"Path\": \"Shared/e2e-folder-a-8805f811\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder A for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-8805f811\",\r\n \"ID\": 2901870,\r\n \"Key\": \"66247ea0-e4ad-4eb5-989d-846926c0acd5\",\r\n \"Path\": \"Shared/e2e-folder-b-8805f811\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-a441329c\",\r\n \"ID\": 2901830,\r\n \"Key\": \"b5eb7093-74cf-443c-8d02-8712f4831c5a\",\r\n \"Path\": \"Shared/e2e-folder-b-a441329c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder B for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-a441329c\",\r\n \"ID\": 2901829,\r\n \"Key\": \"616562ea-0b92-41bc-9c3e-110c3981a647\",\r\n \"Path\": \"Shared/e2e-folder-a-a441329c\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"E2E test folder A for asset sharing\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-a-4c752284\",\r\n \"ID\": 2901827,\r\n \"Key\": \"abd2c82d-ddc3-4c1d-8c9e-6da3a0321921\",\r\n \"Path\": \"Shared/e2e-folder-a-4c752284\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"e2e-folder-b-4c752284\",\r\n \"ID\": 2901826,\r\n \"Key\": \"8a06458d-c42e-4f8b-9295-447bbdf2702e\",\r\n \"Path\": \"Shared/e2e-folder-b-4c752284\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"FinancialPostingFunction\",\r\n \"ID\": 2897454,\r\n \"Key\": \"aa52840c-acc5-44f7-ac5b-9f67f6453ed1\",\r\n \"Path\": \"Shared/uipath-maestro-flow/BillingDispute/FinancialPostingFunction\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"BillingDispute\",\r\n \"ID\": 2897440,\r\n \"Key\": \"79a7d12e-da84-4bf4-ab60-7f34b44dd178\",\r\n \"Path\": \"Shared/uipath-maestro-flow/BillingDispute\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"acme-echo-solution 1\",\r\n \"ID\": 2895681,\r\n \"Key\": \"49f00a14-48be-405d-8568-976dbc6e1594\",\r\n \"Path\": \"Shared/acme-echo-solution 1\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"acme-echo-solution\",\r\n \"ID\": 2895676,\r\n \"Key\": \"726471ba-7358-4d4e-ab6e-980698eccd1c\",\r\n \"Path\": \"Shared/acme-echo-solution\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"uipath-platform-e2e\",\r\n \"ID\": 2892937,\r\n \"Key\": \"c23e5354-4c28-4a8d-ab8e-d63844b35926\",\r\n \"Path\": \"Shared/uipath-platform-e2e\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-18T13:31:50.711Z\",\r\n \"Message\": \"RPA Workflow execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-18T13:31:50.662Z\",\r\n \"Message\": \"Create Ticket: Request failed with error: \\nProviderMessage : code - access_denied, message - You are not authorized to perform this action.\\nProviderErrorCode : 403\\nRequestId : 6a33f346e4b089c6407a42a2\\nMessage : Forbidden\\n\\r\\nStatus code: Forbidden. Error code: DAP-RT-1101.\\nin Main.xaml\\n at ConnectorActivity \\\"Create Ticket\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException\\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendCUDRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-18T13:31:47.931Z\",\r\n \"Message\": \"RPA Workflow execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-18T13:31:50.900Z\",\r\n \"JobId\": 50205597\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-18T13:31:45.143Z\",\r\n \"JobId\": 50205597\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-18T13:31:42.220Z\",\r\n \"JobId\": 50205597\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"38c2b236-c392-420f-9256-b50467e02122\",\r\n \"StartTime\": \"2026-06-18T13:31:45.133Z\",\r\n \"EndTime\": \"2026-06-18T13:31:50.893Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"ResourceOverwrites\": \"{\\\"Connection.29aae689-111d-459f-9a4a-d1ad1d9d2761\\\":{\\\"EntityDisplayName\\\":\\\"uipath\\\",\\\"Level\\\":0,\\\"Properties\\\":{\\\"connectionId\\\":\\\"29aae689-111d-459f-9a4a-d1ad1d9d2761\\\",\\\"folderKey\\\":\\\"b262689d-e2a9-4c1a-8e68-04382a7fff76\\\",\\\"ConnectionId\\\":\\\"29aae689-111d-459f-9a4a-d1ad1d9d2761\\\"}}}\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d014c03c-ed7a-4bd8-b083-d4d59af37049\",\r\n \"Info\": \"Request failed with error: \\nProviderMessage : code - access_denied, message - You are not authorized to perform this action.\\nProviderErrorCode : 403\\nRequestId : 6a33f346e4b089c6407a42a2\\nMessage : Forbidden\\n\\r\\nStatus code: Forbidden. Error code: DAP-RT-1101.\\r\\n\\r\\nin Main.xaml\\n at ConnectorActivity \\\"Create Ticket\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Request failed with error: \\nProviderMessage : code - access_denied, message - You are not authorized to perform this action.\\nProviderErrorCode : 403\\nRequestId : 6a33f346e4b089c6407a42a2\\nMessage : Forbidden\\n\\r\\nStatus code: Forbidden. Error code: DAP-RT-1101.\\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendCUDRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-18T13:31:42.187Z\",\r\n \"ReleaseName\": \"RPA Workflow\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"BBCFBBA7-8063-4F00-E5D1-0B49C871596F\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Serverless\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1088623,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 2976014,\r\n \"FolderKey\": \"e555fecd-20d3-47aa-864c-1c2e588d4769\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": true,\r\n \"LocalSystemAccount\": \"ROBOTUSER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"ServerlessJobType\": \"RobotJob\",\r\n \"LastModificationTime\": \"2026-06-18T13:31:50.893Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"38c2b236c392420f9256b50467e02122\",\r\n \"Id\": 50205597\r\n }\r\n}\r\n" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/task.yaml index 23a04f92f3..3cff40fee0 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-request-failed-dap/task.yaml @@ -15,21 +15,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/README.md new file mode 100644 index 0000000000..a814c63ac4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/README.md @@ -0,0 +1,35 @@ +# rpa-is-response-too-large + +Faithful-replay scenario for the **Integration Service — Response Content Too Large** playbook +([`response-content-too-large.md`](../../../../../skills/uipath-troubleshoot/references/products/integration-service/playbooks/response-content-too-large.md)). + +## What the scenario exercises + +A scheduled unattended job (`Data Export`) faults at a ServiceNow **List Incidents** `ConnectorActivity`. The +runtime throws `UiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Response content too large.` +because an **unbounded list query** returned more than the Integration Service **8 MB JSON limit** will marshal. + +The agent must reach: the provider call *succeeded* and the failure is a **response-size** breach (8 MB JSON limit), +**not** an auth/connection/provider error — then recommend bounding the query (**Max records** + filter / paging), and +passing file bytes outside JSON (1 GB limit) if applicable. + +## Why the two criteria stay aligned (no bypass) + +- The decisive string `Response content too large` lives only in the **job Info / logs**, not the prompt — the agent + must investigate to find it. +- The connection **pings Enabled** and there is **no `ProviderErrorCode`** — a plausible wrong-turn (chase an + auth/connection cause) that a shallow read invites but the evidence rejects. +- The correct diagnosis requires the specific **8 MB limit** + **Max records/filter** fix (and the outside-JSON 1 GB + nuance) from the playbook, so a skill-less shallow answer ("response too large, make it smaller") lands below the + `llm_judge` pass threshold. + +## Mock + +`uip is …` connection/resource queries to canned fixtures; `docsai ask` is passthrough. Evidence surfaces in +`job-get.json` (Info) and `job-logs.json`. + +Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/02e8acc99f.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/02e8acc99f.json deleted file mode 100644 index 6205d04188..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/02e8acc99f.json +++ /dev/null @@ -1 +0,0 @@ -1.195.0 diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/1a3b347f14.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/1a3b347f14.json deleted file mode 100644 index 94b2918533..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/1a3b347f14.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "ProcessList", - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "A1F2E3D4-1122-3344-5566-7788990011AA", - "Name": "Data Export", - "ProcessKey": "DataExport.process.Data.Export", - "ProcessVersion": "1.0.0", - "Description": "", - "IsLatestVersion": true, - "RetentionPeriod": 30, - "RetentionAction": "Delete" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/3db6b96527.json deleted file mode 100644 index c0ce65c9d6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/3db6b96527.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3d4e5f6-3333-4444-5555-666677778888", - "StartTime": "2026-06-30T02:15:41.512Z", - "EndTime": "2026-06-30T02:15:52.004Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "ResourceOverwrites": "{\"Connection.e5f6a7b8-9999-8888-7777-666655554444\":{\"EntityDisplayName\":\"servicenow-prod\",\"Level\":0,\"Properties\":{\"connectionId\":\"e5f6a7b8-9999-8888-7777-666655554444\",\"folderKey\":\"b2c3d4e5-2222-3333-4444-555566667777\",\"ConnectionId\":\"e5f6a7b8-9999-8888-7777-666655554444\"}}}", - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "7b8c9d0e-aa11-bb22-cc33-dd44ee55ff66", - "Info": "Response content too large.\r\n\r\nin Main.xaml\n at ConnectorActivity \"List Incidents\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Response content too large.\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)", - "CreationTime": "2026-06-30T02:15:38.900Z", - "ReleaseName": "Data Export", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "D4E5F6A7-1000-2000-3000-400050006000", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Serverless", - "RequiresUserInteraction": false, - "ReleaseVersionId": 1122334, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 3010001, - "FolderKey": "a1b2c3d4-1111-2222-3333-444455556666", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": true, - "LocalSystemAccount": "ROBOTUSER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "ServerlessJobType": "RobotJob", - "LastModificationTime": "2026-06-30T02:15:52.004Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3d4e5f633334444555566677778888", - "Id": 50301122 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/59c71d8a84.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/59c71d8a84.json deleted file mode 100644 index a27d261ae2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/59c71d8a84.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "e5f6a7b8-9999-8888-7777-666655554444", - "Name": "servicenow-prod", - "ConnectorKey": "uipath-servicenow", - "ConnectorName": "ServiceNow", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "No", - "Folder": "integration-sync", - "FolderKey": "b2c3d4e5-2222-3333-4444-555566667777", - "Created": "2026-05-02T09:11:20.400Z", - "Updated": "2026-05-02T09:11:20.400Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/701433db1c.json deleted file mode 100644 index f501c342a1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/701433db1c.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-06-30T02:15:52.010Z", - "JobId": 50301122 - }, - { - "State": "Running", - "Timestamp": "2026-06-30T02:15:41.520Z", - "JobId": 50301122 - }, - { - "State": "Pending", - "Timestamp": "2026-06-30T02:15:38.930Z", - "JobId": 50301122 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/8d32c56987.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/8d32c56987.json deleted file mode 100644 index a27d261ae2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/8d32c56987.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionList", - "Data": [ - { - "Id": "e5f6a7b8-9999-8888-7777-666655554444", - "Name": "servicenow-prod", - "ConnectorKey": "uipath-servicenow", - "ConnectorName": "ServiceNow", - "State": "Enabled", - "Owner": "original_email@test.com", - "IsDefault": "No", - "Folder": "integration-sync", - "FolderKey": "b2c3d4e5-2222-3333-4444-555566667777", - "Created": "2026-05-02T09:11:20.400Z", - "Updated": "2026-05-02T09:11:20.400Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/README.md deleted file mode 100644 index 52add4f6d7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# rpa-is-response-too-large - -Faithful-replay scenario for the **Integration Service — Response Content Too Large** playbook -([`response-content-too-large.md`](../../../../../skills/uipath-troubleshoot/references/products/integration-service/playbooks/response-content-too-large.md)). - -## What the scenario exercises - -A scheduled unattended job (`Data Export`) faults at a ServiceNow **List Incidents** `ConnectorActivity`. The -runtime throws `UiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Response content too large.` -because an **unbounded list query** returned more than the Integration Service **8 MB JSON limit** will marshal. - -The agent must reach: the provider call *succeeded* and the failure is a **response-size** breach (8 MB JSON limit), -**not** an auth/connection/provider error — then recommend bounding the query (**Max records** + filter / paging), and -passing file bytes outside JSON (1 GB limit) if applicable. - -## Why the two criteria stay aligned (no bypass) - -- The decisive string `Response content too large` lives only in the **job Info / logs**, not the prompt — the agent - must investigate to find it. -- The connection **pings Enabled** and there is **no `ProviderErrorCode`** — a plausible wrong-turn (chase an - auth/connection cause) that a shallow read invites but the evidence rejects. -- The correct diagnosis requires the specific **8 MB limit** + **Max records/filter** fix (and the outside-JSON 1 GB - nuance) from the playbook, so a skill-less shallow answer ("response too large, make it smaller") lands below the - `llm_judge` pass threshold. - -## Mock - -CLI-driven (no `process/` snapshot). `data/m/r/manifest.json` dispatches `uip or …` job/folder/process queries and -`uip is …` connection/resource queries to canned fixtures; `docsai ask` is passthrough. Evidence surfaces in -`job-get.json` (Info) and `job-logs.json`. - -Ground truth for the judge: [`RESOLUTION.md`](./RESOLUTION.md). diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b0ee5b9db3.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b0ee5b9db3.json deleted file mode 100644 index 1dd52dbc1b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b0ee5b9db3.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "ConnectionPing", - "Data": { - "ConnectionId": "e5f6a7b8-9999-8888-7777-666655554444", - "Status": "Enabled", - "Message": "Connection is active and ready for operations." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b58d253fd5.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b58d253fd5.json deleted file mode 100644 index 5e05363e8f..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b58d253fd5.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "Result": "Success", - "Code": "ResourceMetadata", - "Data": { - "name": "incident", - "displayName": "Incident", - "availableOperations": [ - "List", - "Retrieve", - "Create", - "Update", - "Delete" - ], - "files": { - "List": "~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.List.json", - "Retrieve": "~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Retrieve.json", - "Create": "~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Create.json", - "Update": "~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Update.json", - "Delete": "~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Delete.json" - }, - "hint": "Use --operation to see fields, or read the files directly for full details.", - "Warning": "Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields. Run 'is connections list ' to get available connection IDs." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b8d18266dc.json deleted file mode 100644 index 30b6576198..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-06-30T02:15:51.940Z", - "Message": "Data Export execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-30T02:15:51.880Z", - "Message": "List Incidents: Response content too large.\nin Main.xaml\n at ConnectorActivity \"List Incidents\"\n at Sequence \"Sequence\"\n at Main \"Main\"\n\n--- .NET stack trace ---\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Response content too large.\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)" - }, - { - "Level": "Trace", - "TimeStamp": "2026-06-30T02:15:48.220Z", - "Message": "List Incidents: Executing connector operation 'List' on object 'incident'." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-30T02:15:44.010Z", - "Message": "Data Export execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/c3017f84e2.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/c3017f84e2.json deleted file mode 100644 index 2293eb0c92..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/c3017f84e2.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "Result": "Success", - "Code": "ResourceMetadata", - "Data": { - "name": "incident", - "displayName": "List Incidents", - "path": "/table/incident", - "operation": "List", - "metadataFile": "~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.List.json", - "requiredFields": [], - "optionalFields": [ - "sysparm_query", - "sysparm_limit", - "sysparm_offset", - "sysparm_fields", - "sysparm_display_value", - "sysparm_exclude_reference_link", - "maxRecords" - ], - "paginationFields": [ - "sysparm_limit", - "sysparm_offset", - "maxRecords" - ], - "responseFields": [ - "sys_id", - "number", - "short_description", - "description", - "state", - "priority", - "urgency", - "impact", - "category", - "subcategory", - "assigned_to", - "assignment_group", - "caller_id", - "opened_at", - "closed_at", - "resolved_at", - "sys_created_on", - "sys_updated_on", - "comments", - "work_notes", - "close_notes", - "attachment" - ], - "hint": "List returns multiple records. Use 'sysparm_query' to filter and 'sysparm_limit' / 'maxRecords' to bound the number of rows returned.", - "Warning": "Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/ee2628e6b6.json deleted file mode 100644 index b4bbf90acd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Name": "DataExport", - "ID": 3010001, - "Key": "a1b2c3d4-1111-2222-3333-444455556666", - "Path": "Shared/integration-sync/DataExport", - "Type": "Solution", - "Description": "" - }, - { - "Name": "integration-sync", - "ID": 3009980, - "Key": "b2c3d4e5-2222-3333-4444-555566667777", - "Path": "Shared/integration-sync", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Finance", - "ID": 2901921, - "Key": "fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed", - "Path": "Finance", - "Type": "Standard", - "Description": "" - }, - { - "Name": "Shared", - "ID": 2892900, - "Key": "c23e5354-4c28-4a8d-ab8e-d63844b35926", - "Path": "Shared", - "Type": "Standard", - "Description": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/f5f2f217c8.json deleted file mode 100644 index e9299501bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c3d4e5f6-3333-4444-5555-666677778888", - "State": "Faulted", - "ProcessName": "Data Export", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "D4E5F6A7-1000-2000-3000-400050006000", - "StartTime": "2026-06-30T02:15:41.512Z", - "EndTime": "2026-06-30T02:15:52.004Z", - "CreationTime": "2026-06-30T02:15:38.900Z", - "Id": 50301122 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/manifest.json deleted file mode 100644 index cbe8b3e3f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/m/r/manifest.json +++ /dev/null @@ -1,63 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay scenario. Maps uip Orchestrator and Integration Service queries to canned fixtures for a faulted scheduled job. Rules ordered specific-before-generic; docsai ask is passthrough.", - "rules": [ - { - "match": "--version", - "file": "02e8acc99f.json" - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or processes list", - "file": "1a3b347f14.json" - }, - { - "match": "or jobs list --state Faulted", - "file": "f5f2f217c8.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "is connections ping", - "file": "b0ee5b9db3.json" - }, - { - "match": "is connections list --connection-id", - "file": "59c71d8a84.json" - }, - { - "match": "is connections list uipath-servicenow", - "file": "8d32c56987.json" - }, - { - "match": "is resources describe uipath-servicenow incident --operation List", - "file": "c3017f84e2.json" - }, - { - "match": "is resources describe uipath-servicenow incident", - "file": "b58d253fd5.json" - }, - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/uip-fixture.json new file mode 100644 index 0000000000..e34e8c568a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/data/uip-fixture.json @@ -0,0 +1,137 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "is", + "resources", + "describe", + "uipath-servicenow", + "incident", + "--operation", + "List" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ResourceMetadata\",\r\n \"Data\": {\r\n \"name\": \"incident\",\r\n \"displayName\": \"List Incidents\",\r\n \"path\": \"/table/incident\",\r\n \"operation\": \"List\",\r\n \"metadataFile\": \"~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.List.json\",\r\n \"requiredFields\": [],\r\n \"optionalFields\": [\r\n \"sysparm_query\",\r\n \"sysparm_limit\",\r\n \"sysparm_offset\",\r\n \"sysparm_fields\",\r\n \"sysparm_display_value\",\r\n \"sysparm_exclude_reference_link\",\r\n \"maxRecords\"\r\n ],\r\n \"paginationFields\": [\r\n \"sysparm_limit\",\r\n \"sysparm_offset\",\r\n \"maxRecords\"\r\n ],\r\n \"responseFields\": [\r\n \"sys_id\",\r\n \"number\",\r\n \"short_description\",\r\n \"description\",\r\n \"state\",\r\n \"priority\",\r\n \"urgency\",\r\n \"impact\",\r\n \"category\",\r\n \"subcategory\",\r\n \"assigned_to\",\r\n \"assignment_group\",\r\n \"caller_id\",\r\n \"opened_at\",\r\n \"closed_at\",\r\n \"resolved_at\",\r\n \"sys_created_on\",\r\n \"sys_updated_on\",\r\n \"comments\",\r\n \"work_notes\",\r\n \"close_notes\",\r\n \"attachment\"\r\n ],\r\n \"hint\": \"List returns multiple records. Use 'sysparm_query' to filter and 'sysparm_limit' / 'maxRecords' to bound the number of rows returned.\",\r\n \"Warning\": \"Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3d4e5f6-3333-4444-5555-666677778888\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"Data Export\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"D4E5F6A7-1000-2000-3000-400050006000\",\r\n \"StartTime\": \"2026-06-30T02:15:41.512Z\",\r\n \"EndTime\": \"2026-06-30T02:15:52.004Z\",\r\n \"CreationTime\": \"2026-06-30T02:15:38.900Z\",\r\n \"Id\": 50301122\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "resources", + "describe", + "uipath-servicenow", + "incident" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ResourceMetadata\",\r\n \"Data\": {\r\n \"name\": \"incident\",\r\n \"displayName\": \"Incident\",\r\n \"availableOperations\": [\r\n \"List\",\r\n \"Retrieve\",\r\n \"Create\",\r\n \"Update\",\r\n \"Delete\"\r\n ],\r\n \"files\": {\r\n \"List\": \"~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.List.json\",\r\n \"Retrieve\": \"~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Retrieve.json\",\r\n \"Create\": \"~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Create.json\",\r\n \"Update\": \"~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Update.json\",\r\n \"Delete\": \"~/.uipath/cache/integrationservice/uipath-servicenow/_static/incident.Delete.json\"\r\n },\r\n \"hint\": \"Use --operation to see fields, or read the files directly for full details.\",\r\n \"Warning\": \"Results may not include custom fields. Use --connection-id for connection-specific metadata including custom objects and fields. Run 'is connections list ' to get available connection IDs.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "--connection-id" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"e5f6a7b8-9999-8888-7777-666655554444\",\r\n \"Name\": \"servicenow-prod\",\r\n \"ConnectorKey\": \"uipath-servicenow\",\r\n \"ConnectorName\": \"ServiceNow\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"original_email@test.com\",\r\n \"IsDefault\": \"No\",\r\n \"Folder\": \"integration-sync\",\r\n \"FolderKey\": \"b2c3d4e5-2222-3333-4444-555566667777\",\r\n \"Created\": \"2026-05-02T09:11:20.400Z\",\r\n \"Updated\": \"2026-05-02T09:11:20.400Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "list", + "uipath-servicenow" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionList\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"e5f6a7b8-9999-8888-7777-666655554444\",\r\n \"Name\": \"servicenow-prod\",\r\n \"ConnectorKey\": \"uipath-servicenow\",\r\n \"ConnectorName\": \"ServiceNow\",\r\n \"State\": \"Enabled\",\r\n \"Owner\": \"original_email@test.com\",\r\n \"IsDefault\": \"No\",\r\n \"Folder\": \"integration-sync\",\r\n \"FolderKey\": \"b2c3d4e5-2222-3333-4444-555566667777\",\r\n \"Created\": \"2026-05-02T09:11:20.400Z\",\r\n \"Updated\": \"2026-05-02T09:11:20.400Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"DataExport\",\r\n \"ID\": 3010001,\r\n \"Key\": \"a1b2c3d4-1111-2222-3333-444455556666\",\r\n \"Path\": \"Shared/integration-sync/DataExport\",\r\n \"Type\": \"Solution\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"integration-sync\",\r\n \"ID\": 3009980,\r\n \"Key\": \"b2c3d4e5-2222-3333-4444-555566667777\",\r\n \"Path\": \"Shared/integration-sync\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Finance\",\r\n \"ID\": 2901921,\r\n \"Key\": \"fbb6bcb1-11a3-44e4-87b1-9e77b04f25ed\",\r\n \"Path\": \"Finance\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n },\r\n {\r\n \"Name\": \"Shared\",\r\n \"ID\": 2892900,\r\n \"Key\": \"c23e5354-4c28-4a8d-ab8e-d63844b35926\",\r\n \"Path\": \"Shared\",\r\n \"Type\": \"Standard\",\r\n \"Description\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "processes", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ProcessList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"A1F2E3D4-1122-3344-5566-7788990011AA\",\r\n \"Name\": \"Data Export\",\r\n \"ProcessKey\": \"DataExport.process.Data.Export\",\r\n \"ProcessVersion\": \"1.0.0\",\r\n \"Description\": \"\",\r\n \"IsLatestVersion\": true,\r\n \"RetentionPeriod\": 30,\r\n \"RetentionAction\": \"Delete\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3d4e5f6-3333-4444-5555-666677778888\",\r\n \"StartTime\": \"2026-06-30T02:15:41.512Z\",\r\n \"EndTime\": \"2026-06-30T02:15:52.004Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"ResourceOverwrites\": \"{\\\"Connection.e5f6a7b8-9999-8888-7777-666655554444\\\":{\\\"EntityDisplayName\\\":\\\"servicenow-prod\\\",\\\"Level\\\":0,\\\"Properties\\\":{\\\"connectionId\\\":\\\"e5f6a7b8-9999-8888-7777-666655554444\\\",\\\"folderKey\\\":\\\"b2c3d4e5-2222-3333-4444-555566667777\\\",\\\"ConnectionId\\\":\\\"e5f6a7b8-9999-8888-7777-666655554444\\\"}}}\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"7b8c9d0e-aa11-bb22-cc33-dd44ee55ff66\",\r\n \"Info\": \"Response content too large.\\r\\n\\r\\nin Main.xaml\\n at ConnectorActivity \\\"List Incidents\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Response content too large.\\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivityImplementation.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.AsyncTaskCodeActivity.EndExecute(AsyncCodeActivityContext context, IAsyncResult result)\\n at System.Activities.AsyncCodeActivity.CompleteAsyncCodeActivityData.CompleteAsyncCodeActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\r\n \"CreationTime\": \"2026-06-30T02:15:38.900Z\",\r\n \"ReleaseName\": \"Data Export\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"D4E5F6A7-1000-2000-3000-400050006000\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Serverless\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 1122334,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 3010001,\r\n \"FolderKey\": \"a1b2c3d4-1111-2222-3333-444455556666\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": true,\r\n \"LocalSystemAccount\": \"ROBOTUSER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"ServerlessJobType\": \"RobotJob\",\r\n \"LastModificationTime\": \"2026-06-30T02:15:52.004Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3d4e5f633334444555566677778888\",\r\n \"Id\": 50301122\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-30T02:15:51.940Z\",\r\n \"Message\": \"Data Export execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-30T02:15:51.880Z\",\r\n \"Message\": \"List Incidents: Response content too large.\\nin Main.xaml\\n at ConnectorActivity \\\"List Incidents\\\"\\n at Sequence \\\"Sequence\\\"\\n at Main \\\"Main\\\"\\n\\n--- .NET stack trace ---\\nUiPath.IntegrationService.Activities.Runtime.Exceptions.RuntimeException: Response content too large.\\n at UiPath.IntegrationService.Activities.Runtime.Services.RuntimeClientService.SendRequestClientAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, ICollection`1 jitInputs, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Services.ExecutionService.ExecuteOperationAsync(ExecutionParameters executionParameters, ConnectorActivityConfiguration activityConfiguration, Dictionary`2 telemetryUsedProperties, CancellationToken token)\\n at UiPath.IntegrationService.Activities.Runtime.Activities.ConnectorActivity.ExecuteAsync(AsyncCodeActivityContext context, CancellationToken token)\"\r\n },\r\n {\r\n \"Level\": \"Trace\",\r\n \"TimeStamp\": \"2026-06-30T02:15:48.220Z\",\r\n \"Message\": \"List Incidents: Executing connector operation 'List' on object 'incident'.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-30T02:15:44.010Z\",\r\n \"Message\": \"Data Export execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-06-30T02:15:52.010Z\",\r\n \"JobId\": 50301122\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-06-30T02:15:41.520Z\",\r\n \"JobId\": 50301122\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-06-30T02:15:38.930Z\",\r\n \"JobId\": 50301122\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "is", + "connections", + "ping" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ConnectionPing\",\r\n \"Data\": {\r\n \"ConnectionId\": \"e5f6a7b8-9999-8888-7777-666655554444\",\r\n \"Status\": \"Enabled\",\r\n \"Message\": \"Connection is active and ready for operations.\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "--version" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "1.195.0\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/task.yaml b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/task.yaml index 0c35570266..88cee8e18c 100644 --- a/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/integration-service/rpa-is-response-too-large/task.yaml @@ -15,21 +15,12 @@ run_limits: turn_timeout: 3600 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/README.md b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/README.md new file mode 100644 index 0000000000..2095450bbf --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/README.md @@ -0,0 +1,37 @@ +# Maestro Stuck Rpa Job — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +Root Cause: UI selector mismatch in the child automation caused a cascading failure that left the Maestro BPMN instance permanently stuck. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +This scenario is CLI-only — there's no project source, so no `process/` directory is included. The troubleshooting is reproduced entirely from `uip` mock responses. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name maestro-stuck-rpa-job --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/09fdb87f77.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/09fdb87f77.json deleted file mode 100644 index de71ef5437..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/09fdb87f77.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Running", - "Timestamp": "2025-09-23T09:45:59.210Z", - "JobId": 42792329 - }, - { - "State": "Pending", - "Timestamp": "2025-09-23T09:45:55.973Z", - "JobId": 42792329 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/0f1f693bed.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/0f1f693bed.json deleted file mode 100644 index c199ed888d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/0f1f693bed.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "Result": "Success", - "Code": "InstanceIncidents", - "Data": [ - { - "id": "f32ae7e9-fdd4-4978-80a9-101e1496df82", - "tenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "organizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "instanceId": "0fbda085-5c96-4dab-a4b3-012c1914845a", - "runId": "f3a0d71f-663d-4d52-b55b-4c54c7d8eee9", - "elementId": "Activity_EW6HNH", - "elementRunId": "cddd4d8d-3821-441a-8e1c-eda84d570e49", - "folderKey": "36c6d42a-9294-4807-ac48-46781c45fb96", - "processKey": "f9b2ddf0-f912-4c67-b634-b64f7c3ea3a8", - "packageVersion": null, - "incidentId": "cddd4d8d-3821-441a-8e1c-eda84d570e49", - "incidentStatus": "Open", - "incidentType": "System", - "errorCode": "170002", - "errorMessage": "Failure in the Orchestrator Job", - "errorTimeUtc": "2025-09-23T09:47:17.154153Z", - "errorDetails": "{\"Code\":\"Robot\",\"Title\":\"Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[75%] \\n[74%] \\n[73%] \\n[72%] \\n[71%] \\n[70%] \\n[54%] \\n[54%] \\n[54%] \\n[54%] )\",\"Detail\":\"in Main.xaml\\r\\n at NClick \\\"Click 'Collaboration'\\\"\\r\\n at Sequence \\\"Do\\\"\\r\\n at NApplicationCard \\\"Edge My Apps\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\r\\n\\r\\nPossible solutions:\\r\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\r\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\r\\n • If needed, re-indicate the element as its properties might have changed\\r\\n • Use \\\"Check state\\\" activity to check the application state before executing the action\\r\\n • Increase the \\\"Delay before\\\" value to allow time to the application to render entirely and become responsive \\r\\n\\r\\n[Target]: Strict selector failure:\\r\\n\\n\\r\\nThe closest selector matches found are:\\r\\n[75%] \\n[74%] \\n[73%] \\n[72%] \\n[71%] \\n[70%] \\n[54%] \\n[54%] \\n[54%] \\n[54%] )\\r\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\r\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\r\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\r\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n--- End of stack trace from previous location ---\\r\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\r\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\",\"Category\":null,\"Status\":null,\"Timestamp\":null,\"AdditionalProperties\":{}}", - "comment": null, - "userUpdated": null, - "debugMode": "None", - "incidentUpdateTimeUtc": null, - "traceId": null, - "dependentFaultCode": "Robot", - "incidentSeverity": "Error", - "aiSummary": null, - "aiRecommendations": null, - "processType": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/38786cca77.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/38786cca77.json deleted file mode 100644 index 3f80b68688..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/38786cca77.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/3eed399cb5.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/3eed399cb5.json deleted file mode 100644 index fedd97721c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/3eed399cb5.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "Result": "Success", - "Code": "InstanceGet", - "Data": { - "organizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "tenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "instanceId": "0fbda085-5c96-4dab-a4b3-012c1914845a", - "packageKey": "Solution.5.Agentic.Agentic-Process:1.0.0", - "packageId": "Solution.5.Agentic.Agentic-Process", - "packageVersion": "1.0.0", - "latestRunId": "f3a0d71f-663d-4d52-b55b-4c54c7d8eee9", - "latestRunStatus": "Faulted", - "processKey": "f9b2ddf0-f912-4c67-b634-b64f7c3ea3a8", - "folderKey": "36c6d42a-9294-4807-ac48-46781c45fb96", - "userId": 606790, - "instanceDisplayName": "Agentic Process-42792329", - "source": "Manual", - "startedByUser": "Unknown", - "creatorUserKey": "67a356f2-1b63-4f9d-8ada-a1cd385f838b", - "createdTimeUtc": "2025-09-23T09:45:56.8639217Z", - "startedTimeUtc": "2025-09-23T09:46:00.5418245Z", - "completedTimeUtc": null, - "instanceRuns": [ - { - "runId": "f3a0d71f-663d-4d52-b55b-4c54c7d8eee9", - "status": "Faulted", - "startedTimeUtc": "2025-09-23T09:46:00.5418245Z", - "completedTimeUtc": "0001-01-01T00:00:00", - "properties": {} - } - ], - "incidents": null, - "cursors": { - "elementIds": [ - "Activity_EW6HNH" - ] - }, - "externalId": "42792329", - "feedId": "432637b3-71e8-413e-92db-74f74011eda4", - "processType": "ProcessOrchestration", - "licensingContext": null - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/454dae4c2e.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/454dae4c2e.json deleted file mode 100644 index fe51488c70..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/454dae4c2e.json +++ /dev/null @@ -1 +0,0 @@ -[] diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/5c8f93eca3.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/5c8f93eca3.json deleted file mode 100644 index fe624c7067..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/5c8f93eca3.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderListCurrentUser", - "Data": [ - { - "key": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "displayName": "original_email@test.com's workspace", - "fullyQualifiedName": "original_email@test.com's workspace", - "folderType": "Personal", - "id": 1321239 - }, - { - "key": "36c6d42a-9294-4807-ac48-46781c45fb96", - "displayName": "Solution 5", - "fullyQualifiedName": "original_email@test.com's workspace/Solution 5", - "folderType": "Solution", - "parentId": 1321239, - "parentKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "id": 2157426 - }, - { - "key": "ea201852-4965-4a5a-bf8a-9854d3a77275", - "displayName": "Shared", - "fullyQualifiedName": "Shared", - "folderType": "Standard", - "id": 294109 - }, - { - "key": "37f64013-6d26-4942-865b-cec4df0b700f", - "displayName": "Shared2", - "fullyQualifiedName": "Shared2", - "folderType": "Standard", - "id": 951575 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/6690a6b57e.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/6690a6b57e.json deleted file mode 100644 index a54f0b64de..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/6690a6b57e.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "0fbda085-5c96-4dab-a4b3-012c1914845a", - "StartTime": "2025-09-23T09:45:59.153Z", - "State": "Running", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "ResourceOverwrites": "{\"process.solution_folder.BlankProcess6\":{\"EntityDisplayName\":\"BlankProcess6\",\"Level\":0,\"Properties\":{\"folderPath\":\"original_email@test.com's workspace/Solution 5\",\"name\":\"BlankProcess6\"}}}", - "Source": "Manual", - "SourceType": "ProcessOrchestration", - "BatchExecutionKey": "b61d4071-8038-49f5-a45b-68c4b835f215", - "Info": "Job running in workflow-id: 0fbda085-5c96-4dab-a4b3-012c1914845a, attempt:1", - "CreationTime": "2025-09-23T09:45:55.907Z", - "ReleaseName": "Agentic Process", - "Type": "Unattended", - "EnvironmentVariables": "", - "HostMachineName": "", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "ProcessOrchestration", - "RequiresUserInteraction": false, - "ReleaseVersionId": 694032, - "EntryPointPath": "/content/Process.bpmn#Event_start", - "OrganizationUnitId": 2157426, - "FolderKey": "36c6d42a-9294-4807-ac48-46781c45fb96", - "Reference": "", - "ProcessType": "ProcessOrchestration", - "ResumeOnSameContext": false, - "LocalSystemAccount": "uipath\\user1", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "ParentJobKey": "c5a19942-1985-418e-9f71-675018e80449", - "LastModificationTime": "2025-09-23T09:45:59.153Z", - "TraceId": "c5a19942-1985-418e-9f71-675018e80449", - "ParentSpanId": "e44b454a-215e-4de3-8287-92a74ee37185", - "FpsContext": "eyJlbGVtZW50UnVuVG9rZW4iOnsiaW5zdGFuY2VJZCI6ImM1YTE5OTQyLTE5ODUtNDE4ZS05ZjcxLTY3NTAxOGU4MDQ0OS9CcG1uR2VuZXJpY1dvcmtmbG93L0FjdGl2aXR5X0VXNkhOSC9lNDRiNDU0YS0yMTVlLTRkZTMtODI4Ny05MmE3NGVlMzcxODUiLCJydW5JZCI6IjE1OTgwOTVmLTAzMmMtNDE1Ny04YjM1LTM4MzExNWY4YmI1YiIsImVsZW1lbnRJZCI6ImM1YTE5OTQyLTE5ODUtNDE4ZS05ZjcxLTY3NTAxOGU4MDQ0OS9CcG1uR2VuZXJpY1dvcmtmbG93L0FjdGl2aXR5X0VXNkhOSC9lNDRiNDU0YS0yMTVlLTRkZTMtODI4Ny05MmE3NGVlMzcxODUvV2FpdEZvck9yY2hlc3RyYXRvckpvYiJ9LCJQb0NvbnRleHQiOnsiT3JjaGVzdHJhdG9yRm9sZGVyQ29udGV4dCI6eyJGb2xkZXJJZCI6bnVsbCwiRm9sZGVyS2V5IjpudWxsLCJGb2xkZXJQYXRoIjoicmVwbGFjZW1lbnRfZW1haWxAdGVzdC5jb20ncyB3b3Jrc3BhY2UvU29sdXRpb24gNSJ9fX0=", - "Id": 42792329 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/679338ec27.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/679338ec27.json deleted file mode 100644 index 99a7b60bbd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/679338ec27.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error getting job", - "Instructions": "Job with key 'c5a19942-1985-418e-9f71-675018e80449' not found: HTTP 404: Job does not exist." -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/7b3d8c49fd.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/7b3d8c49fd.json deleted file mode 100644 index 5806264efe..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/7b3d8c49fd.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "InstanceAsset", - "Data": "\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n edge_E5vh5n\n \n \n \n \n \n \n \n \n \n \n \n \n \n edge_E5vh5n\n edge_jMlE1f\n \n \n edge_jMlE1f\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n\n" -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/81241cf1e0.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/81241cf1e0.json deleted file mode 100644 index 8bb4293bd6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/81241cf1e0.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "InstanceCursors", - "Data": { - "elementIds": [ - "Activity_EW6HNH" - ] - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/89a8d32282.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/89a8d32282.json deleted file mode 100644 index 62f70f2d58..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/89a8d32282.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/8ac2e392f9.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/8ac2e392f9.json deleted file mode 100644 index 809caff66e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/8ac2e392f9.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "0fbda085-5c96-4dab-a4b3-012c1914845a", - "State": "Running", - "ProcessName": "Agentic Process", - "Source": "ProcessOrchestration", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "StartTime": "2025-09-23T09:45:59.153Z", - "EndTime": "", - "CreationTime": "2025-09-23T09:45:55.907Z", - "Id": 42792329 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/9841bb225e.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/9841bb225e.json deleted file mode 100644 index 186742d632..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/9841bb225e.json +++ /dev/null @@ -1,131 +0,0 @@ -{ - "Result": "Success", - "Code": "TraceSpans", - "Data": [ - { - "Id": "a43877386419d954", - "TraceId": "c5a199421985418e9f71675018e80449", - "ParentId": null, - "Name": "Process Instance: c5a19942-1985-418e-9f71-675018e80449", - "StartTime": "2025-09-23T09:45:52.1294627Z", - "EndTime": "2025-09-23T09:57:40.5808854Z", - "Attributes": "{\"spanType\":\"ProcessRun\",\"status\":\"Cancelled\",\"runId\":\"03496b29-354b-4919-a438-77386419d954\",\"instanceId\":\"c5a19942-1985-418e-9f71-675018e80449\"}", - "Status": 2, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 2, - "SpanType": "ProcessRun", - "ProcessKey": null, - "JobKey": null, - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2025-09-23T09:57:40.7208745Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - }, - { - "Id": "8902277658827b4b", - "TraceId": "c5a199421985418e9f71675018e80449", - "ParentId": "a43877386419d954", - "Name": "Event_start", - "StartTime": "2025-09-23T09:45:52.9761443Z", - "EndTime": "2025-09-23T09:45:53.6476184Z", - "Attributes": "{\"spanType\":\"ElementRun\",\"status\":\"Completed\",\"elementId\":\"Event_start\",\"incomingFlowId\":\"\",\"agentTraceLink\":\"\",\"orchestratorJobLink\":\"\",\"actionCenterTaskLink\":\"\",\"markerItemIndex\":null,\"elementRunId\":\"96de732a-940f-4ed6-8902-277658827b4b\",\"instanceId\":\"c5a19942-1985-418e-9f71-675018e80449\"}", - "Status": 1, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 2, - "SpanType": "ElementRun", - "ProcessKey": null, - "JobKey": null, - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2025-09-23T09:45:53.7746218Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - }, - { - "Id": "828792a74ee37185", - "TraceId": "c5a199421985418e9f71675018e80449", - "ParentId": "a43877386419d954", - "Name": "BlankProcess6", - "StartTime": "2025-09-23T09:45:56.2691758Z", - "EndTime": "2025-09-23T09:57:40.1704937Z", - "Attributes": "{\"spanType\":\"ElementRun\",\"status\":\"Terminated\",\"elementId\":\"Activity_EW6HNH\",\"incomingFlowId\":\"edge_E5vh5n\",\"agentTraceLink\":\"\",\"orchestratorJobLink\":\"\",\"actionCenterTaskLink\":\"\",\"markerItemIndex\":null,\"elementRunId\":\"e44b454a-215e-4de3-8287-92a74ee37185\",\"instanceId\":\"c5a19942-1985-418e-9f71-675018e80449\"}", - "Status": 2, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 2, - "SpanType": "ElementRun", - "ProcessKey": null, - "JobKey": null, - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2025-09-23T09:57:40.341736Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - }, - { - "Id": "93d3ce623175eb84", - "TraceId": "c5a199421985418e9f71675018e80449", - "ParentId": "a43877386419d954", - "Name": "__Action=Canceling__", - "StartTime": "2025-09-23T09:57:39.7637842Z", - "EndTime": "2025-09-23T09:57:39.7637842Z", - "Attributes": "{\"spanType\":\"InstanceOperation\",\"operationType\":\"Cancel\",\"comment\":\"Job type job.stopping in Orchestrator\",\"userId\":\"00000000-0000-0000-0000-000000000000\"}", - "Status": 0, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 2, - "SpanType": "InstanceOperation", - "ProcessKey": null, - "JobKey": null, - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2025-09-23T09:57:39.8086521Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - }, - { - "Id": "9fbf345cce3888ef", - "TraceId": "c5a199421985418e9f71675018e80449", - "ParentId": "a43877386419d954", - "Name": "__Action=Canceling__", - "StartTime": "2025-09-23T09:57:39.9622638Z", - "EndTime": "2025-09-23T09:57:39.9622638Z", - "Attributes": "{\"spanType\":\"InstanceOperation\",\"operationType\":\"Cancel\",\"comment\":\"Job type job.stopping in Orchestrator\",\"userId\":\"00000000-0000-0000-0000-000000000000\"}", - "Status": 0, - "OrganizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "TenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "ExpiryTimeUtc": null, - "FolderKey": "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", - "Source": 2, - "SpanType": "InstanceOperation", - "ProcessKey": null, - "JobKey": null, - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2025-09-23T09:57:39.9889758Z", - "Attachments": null, - "PermissionStatus": 0, - "AgentVersion": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/README.md deleted file mode 100644 index 300ab77af2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# Maestro Stuck Rpa Job — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -Root Cause: UI selector mismatch in the child automation caused a cascading failure that left the Maestro BPMN instance permanently stuck. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -This scenario is CLI-only — there's no project source, so no `process/` directory is included. The troubleshooting is reproduced entirely from `uip` mock responses. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name maestro-stuck-rpa-job --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/bf9f72f2c6.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/bf9f72f2c6.json deleted file mode 100644 index daffe3f753..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/bf9f72f2c6.json +++ /dev/null @@ -1,85 +0,0 @@ -{ - "Result": "Success", - "Code": "InstanceElementExecutions", - "Data": { - "creationUserKey": "67a356f2-1b63-4f9d-8ada-a1cd385f838b", - "folderKey": "36c6d42a-9294-4807-ac48-46781c45fb96", - "instanceDisplayName": "Agentic Process-42792329", - "instanceId": "0fbda085-5c96-4dab-a4b3-012c1914845a", - "organizationId": "27c4be1d-a3fd-4be3-b063-682b88ebee86", - "packageId": "Solution.5.Agentic.Agentic-Process", - "packageKey": "Solution.5.Agentic.Agentic-Process:1.0.0", - "packageVersion": "1.0.0", - "processKey": "f9b2ddf0-f912-4c67-b634-b64f7c3ea3a8", - "source": "Manual", - "tenantId": "6a0f9306-c553-4c37-9d86-7ab06a158774", - "status": "Faulted", - "startedTimeUtc": "2025-09-23T09:46:00.5418245Z", - "completedTimeUtc": null, - "elementExecutions": [ - { - "completedTimeUtc": "2025-09-23T09:46:01.8397458Z", - "elementId": "Event_start", - "elementType": "StartEvent", - "elementExtensionType": null, - "parentRunId": null, - "parentElementId": null, - "parentElementRunId": null, - "runId": "", - "startedTimeUtc": "2025-09-23T09:46:01.3338154Z", - "status": "Completed", - "externalLink": "", - "maestroLink": null, - "elementRuns": [ - { - "status": "Completed", - "startedTimeUtc": "2025-09-23T09:46:01.3338154Z", - "completedTimeUtc": "2025-09-23T09:46:01.8397458Z", - "incomingFlowId": null, - "incomingFlowIds": [], - "elementRunId": "3149376d-cb6c-4c9b-8b50-6e4a41a3d494", - "markerItemIndex": null, - "workflowId": "0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow", - "temporalExecutionId": "0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow/Event_start/3149376d-cb6c-4c9b-8b50-6e4a41a3d494", - "version": 2, - "parentElementRunId": null - } - ], - "jobKey": null - }, - { - "completedTimeUtc": null, - "elementId": "Activity_EW6HNH", - "elementType": "ServiceTask", - "elementExtensionType": "Orchestrator.StartJob", - "parentRunId": null, - "parentElementId": null, - "parentElementRunId": null, - "runId": "", - "startedTimeUtc": "2025-09-23T09:46:04.3956408Z", - "status": "Failed", - "externalLink": "", - "maestroLink": null, - "elementRuns": [ - { - "status": "Failed", - "startedTimeUtc": "2025-09-23T09:46:04.3956408Z", - "completedTimeUtc": "2025-09-23T09:47:16.5310458Z", - "incomingFlowId": "edge_E5vh5n", - "incomingFlowIds": [ - "edge_E5vh5n" - ], - "elementRunId": "cddd4d8d-3821-441a-8e1c-eda84d570e49", - "markerItemIndex": null, - "workflowId": "0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow", - "temporalExecutionId": "0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow/Activity_EW6HNH/cddd4d8d-3821-441a-8e1c-eda84d570e49", - "version": 2, - "parentElementRunId": null - } - ], - "jobKey": null - } - ], - "traceId": "c5a19942-1985-418e-9f71-675018e80449" - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/manifest.json deleted file mode 100644 index 30074c9197..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/m/r/manifest.json +++ /dev/null @@ -1,78 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "maestro bpmn instance asset 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "7b3d8c49fd.json" - }, - { - "match": "or jobs get 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "6690a6b57e.json" - }, - { - "match": "or folders list --output json", - "file": "5c8f93eca3.json" - }, - { - "match": "or jobs history 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "09fdb87f77.json" - }, - { - "match": "or jobs logs 0fbda085-5c96-4dab-a4b3-012c1914845a --level Error --output json", - "file": "38786cca77.json" - }, - { - "match": "maestro bpmn instance get 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "3eed399cb5.json" - }, - { - "match": "maestro bpmn instance incidents 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "0f1f693bed.json" - }, - { - "match": "maestro bpmn instance element-executions 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "bf9f72f2c6.json" - }, - { - "match": "or jobs get c5a19942-1985-418e-9f71-675018e80449 --output json", - "file": "679338ec27.json", - "exit_code": 1 - }, - { - "match": "or jobs list --folder-key 36c6d42a-9294-4807-ac48-46781c45fb96 --created-after 2025-09-23T09:45:00Z --created-before 2025-09-23T09:50:00Z --output json", - "file": "8ac2e392f9.json" - }, - { - "match": "maestro bpmn instance cursors 0fbda085-5c96-4dab-a4b3-012c1914845a --output json", - "file": "81241cf1e0.json" - }, - { - "match": "or jobs list --folder-key 7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0 --created-after 2025-09-23T09:45:00Z --created-before 2025-09-23T09:50:00Z --output json", - "file": "89a8d32282.json" - }, - { - "match": "or jobs logs 0fbda085-5c96-4dab-a4b3-012c1914845a --limit 20 --output json", - "file": "38786cca77.json" - }, - { - "match": "traces spans get --job-key 0fbda085-5c96-4dab-a4b3-012c1914845a", - "file": "9841bb225e.json" - }, - { - "match": "or jobs healing-data 0fbda085-5c96-4dab-a4b3-012c1914845a", - "file": "454dae4c2e.json", - "_doc": "Mocked empty — the original session did not capture HA recovery ZIP for this job." - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` or `passthrough_rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/uip-fixture.json new file mode 100644 index 0000000000..678730c296 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/data/uip-fixture.json @@ -0,0 +1,219 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "36c6d42a-9294-4807-ac48-46781c45fb96", + "--created-after", + "2025-09-23T09:45:00Z", + "--created-before", + "2025-09-23T09:50:00Z", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"0fbda085-5c96-4dab-a4b3-012c1914845a\",\r\n \"State\": \"Running\",\r\n \"ProcessName\": \"Agentic Process\",\r\n \"Source\": \"ProcessOrchestration\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"StartTime\": \"2025-09-23T09:45:59.153Z\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2025-09-23T09:45:55.907Z\",\r\n \"Id\": 42792329\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0", + "--created-after", + "2025-09-23T09:45:00Z", + "--created-before", + "2025-09-23T09:50:00Z", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--limit", + "20", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "maestro", + "bpmn", + "instance", + "asset", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"InstanceAsset\",\r\n \"Data\": \"\\n\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n edge_E5vh5n\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n edge_E5vh5n\\n edge_jMlE1f\\n \\n \\n edge_jMlE1f\\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n\\n\"\r\n}\r\n" + }, + { + "argv": [ + "maestro", + "bpmn", + "instance", + "get", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"InstanceGet\",\r\n \"Data\": {\r\n \"organizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"tenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"instanceId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a\",\r\n \"packageKey\": \"Solution.5.Agentic.Agentic-Process:1.0.0\",\r\n \"packageId\": \"Solution.5.Agentic.Agentic-Process\",\r\n \"packageVersion\": \"1.0.0\",\r\n \"latestRunId\": \"f3a0d71f-663d-4d52-b55b-4c54c7d8eee9\",\r\n \"latestRunStatus\": \"Faulted\",\r\n \"processKey\": \"f9b2ddf0-f912-4c67-b634-b64f7c3ea3a8\",\r\n \"folderKey\": \"36c6d42a-9294-4807-ac48-46781c45fb96\",\r\n \"userId\": 606790,\r\n \"instanceDisplayName\": \"Agentic Process-42792329\",\r\n \"source\": \"Manual\",\r\n \"startedByUser\": \"Unknown\",\r\n \"creatorUserKey\": \"67a356f2-1b63-4f9d-8ada-a1cd385f838b\",\r\n \"createdTimeUtc\": \"2025-09-23T09:45:56.8639217Z\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:00.5418245Z\",\r\n \"completedTimeUtc\": null,\r\n \"instanceRuns\": [\r\n {\r\n \"runId\": \"f3a0d71f-663d-4d52-b55b-4c54c7d8eee9\",\r\n \"status\": \"Faulted\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:00.5418245Z\",\r\n \"completedTimeUtc\": \"0001-01-01T00:00:00\",\r\n \"properties\": {}\r\n }\r\n ],\r\n \"incidents\": null,\r\n \"cursors\": {\r\n \"elementIds\": [\r\n \"Activity_EW6HNH\"\r\n ]\r\n },\r\n \"externalId\": \"42792329\",\r\n \"feedId\": \"432637b3-71e8-413e-92db-74f74011eda4\",\r\n \"processType\": \"ProcessOrchestration\",\r\n \"licensingContext\": null\r\n }\r\n}\r\n" + }, + { + "argv": [ + "maestro", + "bpmn", + "instance", + "incidents", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"InstanceIncidents\",\r\n \"Data\": [\r\n {\r\n \"id\": \"f32ae7e9-fdd4-4978-80a9-101e1496df82\",\r\n \"tenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"organizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"instanceId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a\",\r\n \"runId\": \"f3a0d71f-663d-4d52-b55b-4c54c7d8eee9\",\r\n \"elementId\": \"Activity_EW6HNH\",\r\n \"elementRunId\": \"cddd4d8d-3821-441a-8e1c-eda84d570e49\",\r\n \"folderKey\": \"36c6d42a-9294-4807-ac48-46781c45fb96\",\r\n \"processKey\": \"f9b2ddf0-f912-4c67-b634-b64f7c3ea3a8\",\r\n \"packageVersion\": null,\r\n \"incidentId\": \"cddd4d8d-3821-441a-8e1c-eda84d570e49\",\r\n \"incidentStatus\": \"Open\",\r\n \"incidentType\": \"System\",\r\n \"errorCode\": \"170002\",\r\n \"errorMessage\": \"Failure in the Orchestrator Job\",\r\n \"errorTimeUtc\": \"2025-09-23T09:47:17.154153Z\",\r\n \"errorDetails\": \"{\\\"Code\\\":\\\"Robot\\\",\\\"Title\\\":\\\"Could not find the user-interface (UI) element for this action.\\\\r\\\\n\\\\r\\\\nPossible solutions:\\\\r\\\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\\\r\\\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\\\r\\\\n • If needed, re-indicate the element as its properties might have changed\\\\r\\\\n • Use \\\\\\\"Check state\\\\\\\" activity to check the application state before executing the action\\\\r\\\\n • Increase the \\\\\\\"Delay before\\\\\\\" value to allow time to the application to render entirely and become responsive \\\\r\\\\n\\\\r\\\\n[Target]: Strict selector failure:\\\\r\\\\n\\\\n\\\\r\\\\nThe closest selector matches found are:\\\\r\\\\n[75%] \\\\n[74%] \\\\n[73%] \\\\n[72%] \\\\n[71%] \\\\n[70%] \\\\n[54%] \\\\n[54%] \\\\n[54%] \\\\n[54%] )\\\",\\\"Detail\\\":\\\"in Main.xaml\\\\r\\\\n at NClick \\\\\\\"Click 'Collaboration'\\\\\\\"\\\\r\\\\n at Sequence \\\\\\\"Do\\\\\\\"\\\\r\\\\n at NApplicationCard \\\\\\\"Edge My Apps\\\\\\\"\\\\r\\\\n at Sequence \\\\\\\"Main Sequence\\\\\\\"\\\\r\\\\n at Main \\\\\\\"Main\\\\\\\"\\\\r\\\\n\\\\r\\\\nUiPath.UIAutomationNext.Exceptions.NodeNotFoundException: Could not find the user-interface (UI) element for this action.\\\\r\\\\n\\\\r\\\\nPossible solutions:\\\\r\\\\n • Ensure application is opened and the UI element is visible on the screen at execution time\\\\r\\\\n • Edit the Target of the UI activity and use Validation to debug the issue.\\\\r\\\\n • If needed, re-indicate the element as its properties might have changed\\\\r\\\\n • Use \\\\\\\"Check state\\\\\\\" activity to check the application state before executing the action\\\\r\\\\n • Increase the \\\\\\\"Delay before\\\\\\\" value to allow time to the application to render entirely and become responsive \\\\r\\\\n\\\\r\\\\n[Target]: Strict selector failure:\\\\r\\\\n\\\\n\\\\r\\\\nThe closest selector matches found are:\\\\r\\\\n[75%] \\\\n[74%] \\\\n[73%] \\\\n[72%] \\\\n[71%] \\\\n[70%] \\\\n[54%] \\\\n[54%] \\\\n[54%] \\\\n[54%] )\\\\r\\\\n at UiPath.UIAutomationNext.Activities.Services.RuntimeActivityMonitorService.ActivityFaultedAsync(String activityInstanceId, Exception ex, CancellationToken token)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteActivityAsync(IActivityContextWrapper contextWrapper, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.UIAutomationNext.Activities.RecoverableNativeActivity.ExecuteAsync(NativeActivityContext context, CancellationToken cancellationToken)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeImplementation.BookmarkResumptionCallback(NativeActivityContext context, Object value)\\\\r\\\\n at UiPath.Shared.Activities.AsyncTaskNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at UiPath.Shared.Activities.ContinuableAsyncNativeActivity.BookmarkResumptionCallback(NativeActivityContext context, Bookmark bookmark, Object value)\\\\r\\\\n at System.Activities.Runtime.BookmarkWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n--- End of stack trace from previous location ---\\\\r\\\\n at UiPath.UIAutomationNext.Activities.NApplicationCard.OnFault(NativeActivityFaultContext faultContext, Exception propagatedException, ActivityInstance propagatedFrom)\\\\r\\\\n at System.Activities.Runtime.FaultCallbackWrapper.FaultWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)\\\",\\\"Category\\\":null,\\\"Status\\\":null,\\\"Timestamp\\\":null,\\\"AdditionalProperties\\\":{}}\",\r\n \"comment\": null,\r\n \"userUpdated\": null,\r\n \"debugMode\": \"None\",\r\n \"incidentUpdateTimeUtc\": null,\r\n \"traceId\": null,\r\n \"dependentFaultCode\": \"Robot\",\r\n \"incidentSeverity\": \"Error\",\r\n \"aiSummary\": null,\r\n \"aiRecommendations\": null,\r\n \"processType\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "maestro", + "bpmn", + "instance", + "element-executions", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"InstanceElementExecutions\",\r\n \"Data\": {\r\n \"creationUserKey\": \"67a356f2-1b63-4f9d-8ada-a1cd385f838b\",\r\n \"folderKey\": \"36c6d42a-9294-4807-ac48-46781c45fb96\",\r\n \"instanceDisplayName\": \"Agentic Process-42792329\",\r\n \"instanceId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a\",\r\n \"organizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"packageId\": \"Solution.5.Agentic.Agentic-Process\",\r\n \"packageKey\": \"Solution.5.Agentic.Agentic-Process:1.0.0\",\r\n \"packageVersion\": \"1.0.0\",\r\n \"processKey\": \"f9b2ddf0-f912-4c67-b634-b64f7c3ea3a8\",\r\n \"source\": \"Manual\",\r\n \"tenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"status\": \"Faulted\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:00.5418245Z\",\r\n \"completedTimeUtc\": null,\r\n \"elementExecutions\": [\r\n {\r\n \"completedTimeUtc\": \"2025-09-23T09:46:01.8397458Z\",\r\n \"elementId\": \"Event_start\",\r\n \"elementType\": \"StartEvent\",\r\n \"elementExtensionType\": null,\r\n \"parentRunId\": null,\r\n \"parentElementId\": null,\r\n \"parentElementRunId\": null,\r\n \"runId\": \"\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:01.3338154Z\",\r\n \"status\": \"Completed\",\r\n \"externalLink\": \"\",\r\n \"maestroLink\": null,\r\n \"elementRuns\": [\r\n {\r\n \"status\": \"Completed\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:01.3338154Z\",\r\n \"completedTimeUtc\": \"2025-09-23T09:46:01.8397458Z\",\r\n \"incomingFlowId\": null,\r\n \"incomingFlowIds\": [],\r\n \"elementRunId\": \"3149376d-cb6c-4c9b-8b50-6e4a41a3d494\",\r\n \"markerItemIndex\": null,\r\n \"workflowId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow\",\r\n \"temporalExecutionId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow/Event_start/3149376d-cb6c-4c9b-8b50-6e4a41a3d494\",\r\n \"version\": 2,\r\n \"parentElementRunId\": null\r\n }\r\n ],\r\n \"jobKey\": null\r\n },\r\n {\r\n \"completedTimeUtc\": null,\r\n \"elementId\": \"Activity_EW6HNH\",\r\n \"elementType\": \"ServiceTask\",\r\n \"elementExtensionType\": \"Orchestrator.StartJob\",\r\n \"parentRunId\": null,\r\n \"parentElementId\": null,\r\n \"parentElementRunId\": null,\r\n \"runId\": \"\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:04.3956408Z\",\r\n \"status\": \"Failed\",\r\n \"externalLink\": \"\",\r\n \"maestroLink\": null,\r\n \"elementRuns\": [\r\n {\r\n \"status\": \"Failed\",\r\n \"startedTimeUtc\": \"2025-09-23T09:46:04.3956408Z\",\r\n \"completedTimeUtc\": \"2025-09-23T09:47:16.5310458Z\",\r\n \"incomingFlowId\": \"edge_E5vh5n\",\r\n \"incomingFlowIds\": [\r\n \"edge_E5vh5n\"\r\n ],\r\n \"elementRunId\": \"cddd4d8d-3821-441a-8e1c-eda84d570e49\",\r\n \"markerItemIndex\": null,\r\n \"workflowId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow\",\r\n \"temporalExecutionId\": \"0fbda085-5c96-4dab-a4b3-012c1914845a/BpmnGenericWorkflow/Activity_EW6HNH/cddd4d8d-3821-441a-8e1c-eda84d570e49\",\r\n \"version\": 2,\r\n \"parentElementRunId\": null\r\n }\r\n ],\r\n \"jobKey\": null\r\n }\r\n ],\r\n \"traceId\": \"c5a19942-1985-418e-9f71-675018e80449\"\r\n }\r\n}\r\n" + }, + { + "argv": [ + "maestro", + "bpmn", + "instance", + "cursors", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"InstanceCursors\",\r\n \"Data\": {\r\n \"elementIds\": [\r\n \"Activity_EW6HNH\"\r\n ]\r\n }\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "0fbda085-5c96-4dab-a4b3-012c1914845a" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"TraceSpans\",\r\n \"Data\": [\r\n {\r\n \"Id\": \"a43877386419d954\",\r\n \"TraceId\": \"c5a199421985418e9f71675018e80449\",\r\n \"ParentId\": null,\r\n \"Name\": \"Process Instance: c5a19942-1985-418e-9f71-675018e80449\",\r\n \"StartTime\": \"2025-09-23T09:45:52.1294627Z\",\r\n \"EndTime\": \"2025-09-23T09:57:40.5808854Z\",\r\n \"Attributes\": \"{\\\"spanType\\\":\\\"ProcessRun\\\",\\\"status\\\":\\\"Cancelled\\\",\\\"runId\\\":\\\"03496b29-354b-4919-a438-77386419d954\\\",\\\"instanceId\\\":\\\"c5a19942-1985-418e-9f71-675018e80449\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 2,\r\n \"SpanType\": \"ProcessRun\",\r\n \"ProcessKey\": null,\r\n \"JobKey\": null,\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2025-09-23T09:57:40.7208745Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n },\r\n {\r\n \"Id\": \"8902277658827b4b\",\r\n \"TraceId\": \"c5a199421985418e9f71675018e80449\",\r\n \"ParentId\": \"a43877386419d954\",\r\n \"Name\": \"Event_start\",\r\n \"StartTime\": \"2025-09-23T09:45:52.9761443Z\",\r\n \"EndTime\": \"2025-09-23T09:45:53.6476184Z\",\r\n \"Attributes\": \"{\\\"spanType\\\":\\\"ElementRun\\\",\\\"status\\\":\\\"Completed\\\",\\\"elementId\\\":\\\"Event_start\\\",\\\"incomingFlowId\\\":\\\"\\\",\\\"agentTraceLink\\\":\\\"\\\",\\\"orchestratorJobLink\\\":\\\"\\\",\\\"actionCenterTaskLink\\\":\\\"\\\",\\\"markerItemIndex\\\":null,\\\"elementRunId\\\":\\\"96de732a-940f-4ed6-8902-277658827b4b\\\",\\\"instanceId\\\":\\\"c5a19942-1985-418e-9f71-675018e80449\\\"}\",\r\n \"Status\": 1,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 2,\r\n \"SpanType\": \"ElementRun\",\r\n \"ProcessKey\": null,\r\n \"JobKey\": null,\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2025-09-23T09:45:53.7746218Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n },\r\n {\r\n \"Id\": \"828792a74ee37185\",\r\n \"TraceId\": \"c5a199421985418e9f71675018e80449\",\r\n \"ParentId\": \"a43877386419d954\",\r\n \"Name\": \"BlankProcess6\",\r\n \"StartTime\": \"2025-09-23T09:45:56.2691758Z\",\r\n \"EndTime\": \"2025-09-23T09:57:40.1704937Z\",\r\n \"Attributes\": \"{\\\"spanType\\\":\\\"ElementRun\\\",\\\"status\\\":\\\"Terminated\\\",\\\"elementId\\\":\\\"Activity_EW6HNH\\\",\\\"incomingFlowId\\\":\\\"edge_E5vh5n\\\",\\\"agentTraceLink\\\":\\\"\\\",\\\"orchestratorJobLink\\\":\\\"\\\",\\\"actionCenterTaskLink\\\":\\\"\\\",\\\"markerItemIndex\\\":null,\\\"elementRunId\\\":\\\"e44b454a-215e-4de3-8287-92a74ee37185\\\",\\\"instanceId\\\":\\\"c5a19942-1985-418e-9f71-675018e80449\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 2,\r\n \"SpanType\": \"ElementRun\",\r\n \"ProcessKey\": null,\r\n \"JobKey\": null,\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2025-09-23T09:57:40.341736Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n },\r\n {\r\n \"Id\": \"93d3ce623175eb84\",\r\n \"TraceId\": \"c5a199421985418e9f71675018e80449\",\r\n \"ParentId\": \"a43877386419d954\",\r\n \"Name\": \"__Action=Canceling__\",\r\n \"StartTime\": \"2025-09-23T09:57:39.7637842Z\",\r\n \"EndTime\": \"2025-09-23T09:57:39.7637842Z\",\r\n \"Attributes\": \"{\\\"spanType\\\":\\\"InstanceOperation\\\",\\\"operationType\\\":\\\"Cancel\\\",\\\"comment\\\":\\\"Job type job.stopping in Orchestrator\\\",\\\"userId\\\":\\\"00000000-0000-0000-0000-000000000000\\\"}\",\r\n \"Status\": 0,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 2,\r\n \"SpanType\": \"InstanceOperation\",\r\n \"ProcessKey\": null,\r\n \"JobKey\": null,\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2025-09-23T09:57:39.8086521Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n },\r\n {\r\n \"Id\": \"9fbf345cce3888ef\",\r\n \"TraceId\": \"c5a199421985418e9f71675018e80449\",\r\n \"ParentId\": \"a43877386419d954\",\r\n \"Name\": \"__Action=Canceling__\",\r\n \"StartTime\": \"2025-09-23T09:57:39.9622638Z\",\r\n \"EndTime\": \"2025-09-23T09:57:39.9622638Z\",\r\n \"Attributes\": \"{\\\"spanType\\\":\\\"InstanceOperation\\\",\\\"operationType\\\":\\\"Cancel\\\",\\\"comment\\\":\\\"Job type job.stopping in Orchestrator\\\",\\\"userId\\\":\\\"00000000-0000-0000-0000-000000000000\\\"}\",\r\n \"Status\": 0,\r\n \"OrganizationId\": \"27c4be1d-a3fd-4be3-b063-682b88ebee86\",\r\n \"TenantId\": \"6a0f9306-c553-4c37-9d86-7ab06a158774\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"Source\": 2,\r\n \"SpanType\": \"InstanceOperation\",\r\n \"ProcessKey\": null,\r\n \"JobKey\": null,\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2025-09-23T09:57:39.9889758Z\",\r\n \"Attachments\": null,\r\n \"PermissionStatus\": 0,\r\n \"AgentVersion\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"0fbda085-5c96-4dab-a4b3-012c1914845a\",\r\n \"StartTime\": \"2025-09-23T09:45:59.153Z\",\r\n \"State\": \"Running\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"ResourceOverwrites\": \"{\\\"process.solution_folder.BlankProcess6\\\":{\\\"EntityDisplayName\\\":\\\"BlankProcess6\\\",\\\"Level\\\":0,\\\"Properties\\\":{\\\"folderPath\\\":\\\"original_email@test.com's workspace/Solution 5\\\",\\\"name\\\":\\\"BlankProcess6\\\"}}}\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"ProcessOrchestration\",\r\n \"BatchExecutionKey\": \"b61d4071-8038-49f5-a45b-68c4b835f215\",\r\n \"Info\": \"Job running in workflow-id: 0fbda085-5c96-4dab-a4b3-012c1914845a, attempt:1\",\r\n \"CreationTime\": \"2025-09-23T09:45:55.907Z\",\r\n \"ReleaseName\": \"Agentic Process\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"HostMachineName\": \"\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"ProcessOrchestration\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 694032,\r\n \"EntryPointPath\": \"/content/Process.bpmn#Event_start\",\r\n \"OrganizationUnitId\": 2157426,\r\n \"FolderKey\": \"36c6d42a-9294-4807-ac48-46781c45fb96\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"ProcessOrchestration\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"uipath\\\\user1\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"ParentJobKey\": \"c5a19942-1985-418e-9f71-675018e80449\",\r\n \"LastModificationTime\": \"2025-09-23T09:45:59.153Z\",\r\n \"TraceId\": \"c5a19942-1985-418e-9f71-675018e80449\",\r\n \"ParentSpanId\": \"e44b454a-215e-4de3-8287-92a74ee37185\",\r\n \"FpsContext\": \"eyJlbGVtZW50UnVuVG9rZW4iOnsiaW5zdGFuY2VJZCI6ImM1YTE5OTQyLTE5ODUtNDE4ZS05ZjcxLTY3NTAxOGU4MDQ0OS9CcG1uR2VuZXJpY1dvcmtmbG93L0FjdGl2aXR5X0VXNkhOSC9lNDRiNDU0YS0yMTVlLTRkZTMtODI4Ny05MmE3NGVlMzcxODUiLCJydW5JZCI6IjE1OTgwOTVmLTAzMmMtNDE1Ny04YjM1LTM4MzExNWY4YmI1YiIsImVsZW1lbnRJZCI6ImM1YTE5OTQyLTE5ODUtNDE4ZS05ZjcxLTY3NTAxOGU4MDQ0OS9CcG1uR2VuZXJpY1dvcmtmbG93L0FjdGl2aXR5X0VXNkhOSC9lNDRiNDU0YS0yMTVlLTRkZTMtODI4Ny05MmE3NGVlMzcxODUvV2FpdEZvck9yY2hlc3RyYXRvckpvYiJ9LCJQb0NvbnRleHQiOnsiT3JjaGVzdHJhdG9yRm9sZGVyQ29udGV4dCI6eyJGb2xkZXJJZCI6bnVsbCwiRm9sZGVyS2V5IjpudWxsLCJGb2xkZXJQYXRoIjoicmVwbGFjZW1lbnRfZW1haWxAdGVzdC5jb20ncyB3b3Jrc3BhY2UvU29sdXRpb24gNSJ9fX0=\",\r\n \"Id\": 42792329\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "0fbda085-5c96-4dab-a4b3-012c1914845a", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2025-09-23T09:45:59.210Z\",\r\n \"JobId\": 42792329\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2025-09-23T09:45:55.973Z\",\r\n \"JobId\": 42792329\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c5a19942-1985-418e-9f71-675018e80449", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error getting job\",\r\n \"Instructions\": \"Job with key 'c5a19942-1985-418e-9f71-675018e80449' not found: HTTP 404: Job does not exist.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "healing-data", + "0fbda085-5c96-4dab-a4b3-012c1914845a" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "[]\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderListCurrentUser\",\r\n \"Data\": [\r\n {\r\n \"key\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"displayName\": \"original_email@test.com's workspace\",\r\n \"fullyQualifiedName\": \"original_email@test.com's workspace\",\r\n \"folderType\": \"Personal\",\r\n \"id\": 1321239\r\n },\r\n {\r\n \"key\": \"36c6d42a-9294-4807-ac48-46781c45fb96\",\r\n \"displayName\": \"Solution 5\",\r\n \"fullyQualifiedName\": \"original_email@test.com's workspace/Solution 5\",\r\n \"folderType\": \"Solution\",\r\n \"parentId\": 1321239,\r\n \"parentKey\": \"7b6f4886-bec5-4ffc-91cc-dd9a3f9febd0\",\r\n \"id\": 2157426\r\n },\r\n {\r\n \"key\": \"ea201852-4965-4a5a-bf8a-9854d3a77275\",\r\n \"displayName\": \"Shared\",\r\n \"fullyQualifiedName\": \"Shared\",\r\n \"folderType\": \"Standard\",\r\n \"id\": 294109\r\n },\r\n {\r\n \"key\": \"37f64013-6d26-4942-865b-cec4df0b700f\",\r\n \"displayName\": \"Shared2\",\r\n \"fullyQualifiedName\": \"Shared2\",\r\n \"folderType\": \"Standard\",\r\n \"id\": 951575\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/task.yaml b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/task.yaml index 06b99e6359..ce2107bc56 100644 --- a/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/maestro/maestro-stuck-rpa-job/task.yaml @@ -17,21 +17,12 @@ run_limits: turn_timeout: 2700 sandbox: - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/README.md new file mode 100644 index 0000000000..d2e6153ba3 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/README.md @@ -0,0 +1,55 @@ +# Credential Store Unavailable + +Reproduces the `credential-store-unavailable` playbook (Orchestrator +Database store branch): Orchestrator cannot retrieve the robot +credential because the store backend is unreachable, so jobs fail to +start. + +``` +Unable to retrieve credentials from Orchestrator Database credential +store. Please check your connection settings and ensure the +Orchestrator Database service is running. +``` + +## What this scenario uncovers + +**Root Cause:** The Orchestrator SQL database backing the +"Orchestrator Database" credential store is unreachable (connection +timeout). The credential is never retrieved, so no logon is +attempted. Two unrelated processes (`SecurePortalBot`, +`VendorSyncBot`) on different machines fail identically — store-wide. +Fix: restore Orchestrator DB / credential-store connectivity, then +rerun. + +Maps to: +`references/products/orchestrator/playbooks/credential-store-unavailable.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal unattended UiPath project | + +> Fixtures authored from the playbook signature, not captured from a +> real session. + +## Distinguishing fingerprint + +The error is a credential **retrieval** failure (no logon code, no +`0x0000052E`), and it is **store-wide** (two unrelated processes on +different machines). That rules out a wrong/expired password +(robot-credentials / logon-failure) and points at the store backend. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the credential-store backend (Orchestrator SQL + database) as unreachable — not a wrong password — and recommended + restoring store/DB connectivity (and testing the store) before + rerunning. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/3bca59ffbc.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/3bca59ffbc.json deleted file mode 100644 index cc555d0a0c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/3bca59ffbc.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-27T05:00:02.740Z", "Message": "Job 'aabbccdd-1122-3344-5566-778899aabbcc' failed to start. Unable to retrieve credentials from Orchestrator Database credential store. Please check your connection settings and ensure the Orchestrator Database service is running."}, - {"Level": "Error", "TimeStamp": "2026-06-27T05:00:02.500Z", "Message": "[Orchestrator] Credential store 'Orchestrator Database' read failed: unable to open a connection to the Orchestrator SQL database (timeout). No credential was retrieved; Windows logon was not attempted."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/README.md deleted file mode 100644 index 509aa67526..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Credential Store Unavailable - -Reproduces the `credential-store-unavailable` playbook (Orchestrator -Database store branch): Orchestrator cannot retrieve the robot -credential because the store backend is unreachable, so jobs fail to -start. - -``` -Unable to retrieve credentials from Orchestrator Database credential -store. Please check your connection settings and ensure the -Orchestrator Database service is running. -``` - -## What this scenario uncovers - -**Root Cause:** The Orchestrator SQL database backing the -"Orchestrator Database" credential store is unreachable (connection -timeout). The credential is never retrieved, so no logon is -attempted. Two unrelated processes (`SecurePortalBot`, -`VendorSyncBot`) on different machines fail identically — store-wide. -Fix: restore Orchestrator DB / credential-store connectivity, then -rerun. - -Maps to: -`references/products/orchestrator/playbooks/credential-store-unavailable.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal unattended UiPath project | -| `data/m/r/*.json` | **synthetic** canned `uip` responses (jobs get/list/logs) showing the store read failing on a SQL connection timeout | -| `data/m/r/manifest.json` | dispatch table | - -> Fixtures authored from the playbook signature, not captured from a -> real session. - -## Distinguishing fingerprint - -The error is a credential **retrieval** failure (no logon code, no -`0x0000052E`), and it is **store-wide** (two unrelated processes on -different machines). That rules out a wrong/expired password -(robot-credentials / logon-failure) and points at the store backend. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the credential-store backend (Orchestrator SQL - database) as unreachable — not a wrong password — and recommended - restoring store/DB connectivity (and testing the store) before - rerunning. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/b7c928630b.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/b7c928630b.json deleted file mode 100644 index cc555d0a0c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/b7c928630b.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-27T05:00:02.740Z", "Message": "Job 'aabbccdd-1122-3344-5566-778899aabbcc' failed to start. Unable to retrieve credentials from Orchestrator Database credential store. Please check your connection settings and ensure the Orchestrator Database service is running."}, - {"Level": "Error", "TimeStamp": "2026-06-27T05:00:02.500Z", "Message": "[Orchestrator] Credential store 'Orchestrator Database' read failed: unable to open a connection to the Orchestrator SQL database (timeout). No credential was retrieved; Windows logon was not attempted."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/baee8dcd63.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/baee8dcd63.json deleted file mode 100644 index 8812c617f4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/baee8dcd63.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aabbccdd-1122-3344-5566-778899aabbcc", - "StartTime": "2026-06-27T05:00:02.100Z", - "EndTime": "2026-06-27T05:00:02.760Z", - "State": "Faulted", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Unable to retrieve credentials from Orchestrator Database credential store. Please check your connection settings and ensure the Orchestrator Database service is running.", - "CreationTime": "2026-06-27T05:00:02.000Z", - "ReleaseName": "SecurePortalBot", - "Type": "Unattended", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "OrganizationUnitId": 7791640, - "FolderKey": "e5c5d6e7-f8a9-0123-4567-89abcdef1234", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\SVCPORTAL", - "LastModificationTime": "2026-06-27T05:00:02.760Z", - "JobError": {}, - "ErrorCode": "Orchestrator", - "TraceId": "aabbccdd-1122-3344-5566-778899aabbcc", - "Id": 671880011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/d87aa5fe36.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/d87aa5fe36.json deleted file mode 100644 index 4cba8390d4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/d87aa5fe36.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - {"Key": "aabbccdd-1122-3344-5566-778899aabbcc", "State": "Faulted", "ProcessName": "SecurePortalBot", "Source": "Schedule", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST", "StartTime": "2026-06-27T05:00:02.100Z", "EndTime": "2026-06-27T05:00:02.760Z", "CreationTime": "2026-06-27T05:00:02.000Z"}, - {"Key": "aabbccdd-1122-3344-5566-778899aabbdd", "State": "Faulted", "ProcessName": "VendorSyncBot", "Source": "Schedule", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST-2", "StartTime": "2026-06-27T05:00:41.320Z", "EndTime": "2026-06-27T05:00:41.980Z", "CreationTime": "2026-06-27T05:00:41.200Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/ee2628e6b6.json deleted file mode 100644 index c7268093dd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - {"Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", "Name": "original_email@test.com's workspace", "Path": "original_email@test.com's workspace", "Type": "Personal", "ParentKey": ""}, - {"Key": "e5c5d6e7-f8a9-0123-4567-89abcdef1234", "Name": "SecureOps", "Path": "SecureOps", "Description": "Folder hosting portal automations that use the Orchestrator credential store.", "Type": "Standard", "ParentKey": ""}, - {"Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", "Name": "Shared", "Path": "Shared", "Type": "Standard", "ParentKey": ""} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/f5f2f217c8.json deleted file mode 100644 index 4cba8390d4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - {"Key": "aabbccdd-1122-3344-5566-778899aabbcc", "State": "Faulted", "ProcessName": "SecurePortalBot", "Source": "Schedule", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST", "StartTime": "2026-06-27T05:00:02.100Z", "EndTime": "2026-06-27T05:00:02.760Z", "CreationTime": "2026-06-27T05:00:02.000Z"}, - {"Key": "aabbccdd-1122-3344-5566-778899aabbdd", "State": "Faulted", "ProcessName": "VendorSyncBot", "Source": "Schedule", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST-2", "StartTime": "2026-06-27T05:00:41.320Z", "EndTime": "2026-06-27T05:00:41.980Z", "CreationTime": "2026-06-27T05:00:41.200Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/manifest.json deleted file mode 100644 index cdbadaa477..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/m/r/manifest.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the credential-store-unavailable scenario (credential-store-unavailable playbook, Orchestrator Database store branch). The agent chains: folders list -> jobs list --state Faulted (two processes, same store error = store-wide) -> jobs get (Info: retrieval from Orchestrator Database credential store, no logon code) -> jobs logs (SQL DB unreachable; no logon attempted). Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key e5c5d6e7-f8a9-0123-4567-89abcdef1234 --state Faulted", "file": "d87aa5fe36.json" }, - { "match": "or jobs list --state Faulted", "file": "f5f2f217c8.json" }, - { "match": "or jobs get aabbccdd-1122-3344-5566-778899aabbcc", "file": "baee8dcd63.json" }, - { "match": "or jobs logs aabbccdd-1122-3344-5566-778899aabbcc --level Error", "file": "b7c928630b.json" }, - { "match": "or jobs logs aabbccdd-1122-3344-5566-778899aabbcc", "file": "3bca59ffbc.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0 } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/uip-fixture.json new file mode 100644 index 0000000000..b3b3902c90 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/data/uip-fixture.json @@ -0,0 +1,80 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e5c5d6e7-f8a9-0123-4567-89abcdef1234", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\"Key\": \"aabbccdd-1122-3344-5566-778899aabbcc\", \"State\": \"Faulted\", \"ProcessName\": \"SecurePortalBot\", \"Source\": \"Schedule\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST\", \"StartTime\": \"2026-06-27T05:00:02.100Z\", \"EndTime\": \"2026-06-27T05:00:02.760Z\", \"CreationTime\": \"2026-06-27T05:00:02.000Z\"},\r\n {\"Key\": \"aabbccdd-1122-3344-5566-778899aabbdd\", \"State\": \"Faulted\", \"ProcessName\": \"VendorSyncBot\", \"Source\": \"Schedule\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST-2\", \"StartTime\": \"2026-06-27T05:00:41.320Z\", \"EndTime\": \"2026-06-27T05:00:41.980Z\", \"CreationTime\": \"2026-06-27T05:00:41.200Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aabbccdd-1122-3344-5566-778899aabbcc", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-27T05:00:02.740Z\", \"Message\": \"Job 'aabbccdd-1122-3344-5566-778899aabbcc' failed to start. Unable to retrieve credentials from Orchestrator Database credential store. Please check your connection settings and ensure the Orchestrator Database service is running.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-27T05:00:02.500Z\", \"Message\": \"[Orchestrator] Credential store 'Orchestrator Database' read failed: unable to open a connection to the Orchestrator SQL database (timeout). No credential was retrieved; Windows logon was not attempted.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\"Key\": \"aabbccdd-1122-3344-5566-778899aabbcc\", \"State\": \"Faulted\", \"ProcessName\": \"SecurePortalBot\", \"Source\": \"Schedule\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST\", \"StartTime\": \"2026-06-27T05:00:02.100Z\", \"EndTime\": \"2026-06-27T05:00:02.760Z\", \"CreationTime\": \"2026-06-27T05:00:02.000Z\"},\r\n {\"Key\": \"aabbccdd-1122-3344-5566-778899aabbdd\", \"State\": \"Faulted\", \"ProcessName\": \"VendorSyncBot\", \"Source\": \"Schedule\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST-2\", \"StartTime\": \"2026-06-27T05:00:41.320Z\", \"EndTime\": \"2026-06-27T05:00:41.980Z\", \"CreationTime\": \"2026-06-27T05:00:41.200Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aabbccdd-1122-3344-5566-778899aabbcc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aabbccdd-1122-3344-5566-778899aabbcc\",\r\n \"StartTime\": \"2026-06-27T05:00:02.100Z\",\r\n \"EndTime\": \"2026-06-27T05:00:02.760Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Unable to retrieve credentials from Orchestrator Database credential store. Please check your connection settings and ensure the Orchestrator Database service is running.\",\r\n \"CreationTime\": \"2026-06-27T05:00:02.000Z\",\r\n \"ReleaseName\": \"SecurePortalBot\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"OrganizationUnitId\": 7791640,\r\n \"FolderKey\": \"e5c5d6e7-f8a9-0123-4567-89abcdef1234\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVCPORTAL\",\r\n \"LastModificationTime\": \"2026-06-27T05:00:02.760Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Orchestrator\",\r\n \"TraceId\": \"aabbccdd-1122-3344-5566-778899aabbcc\",\r\n \"Id\": 671880011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aabbccdd-1122-3344-5566-778899aabbcc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-27T05:00:02.740Z\", \"Message\": \"Job 'aabbccdd-1122-3344-5566-778899aabbcc' failed to start. Unable to retrieve credentials from Orchestrator Database credential store. Please check your connection settings and ensure the Orchestrator Database service is running.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-27T05:00:02.500Z\", \"Message\": \"[Orchestrator] Credential store 'Orchestrator Database' read failed: unable to open a connection to the Orchestrator SQL database (timeout). No credential was retrieved; Windows logon was not attempted.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\", \"Name\": \"original_email@test.com's workspace\", \"Path\": \"original_email@test.com's workspace\", \"Type\": \"Personal\", \"ParentKey\": \"\"},\r\n {\"Key\": \"e5c5d6e7-f8a9-0123-4567-89abcdef1234\", \"Name\": \"SecureOps\", \"Path\": \"SecureOps\", \"Description\": \"Folder hosting portal automations that use the Orchestrator credential store.\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\", \"Name\": \"Shared\", \"Path\": \"Shared\", \"Type\": \"Standard\", \"ParentKey\": \"\"}\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/task.yaml index 909a810555..28ac5569c8 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/credential-store-unavailable/task.yaml @@ -28,21 +28,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/README.md new file mode 100644 index 0000000000..100a3dcbe9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/README.md @@ -0,0 +1,88 @@ +# Logon Failure — Password Mismatch (Stale Orchestrator Credential) + +This scenario reproduces the **"Could Not Start Executor — Logon +Failed for User"** playbook's **branch 3**: the AD password was +rotated for the unattended robot user, but the credential stored in +Orchestrator was not updated to match. Every job since the rotation +faults with: + +``` +Could not start executor. Logon failed for user UIPATH\USER1. +The user name or password is incorrect. (0x0000052E). +Last error: 131092. RDP connection failed. +``` + +## What this scenario uncovers + +**Root Cause:** Three identical `InvoiceIngestor` runs (keys +`aabb1099-...`, `aabb1100-...`, `aabb1122-...`) faulted within +~25 minutes for service account `UIPATH\USER1` on `MOCK-HOST`. AD +`PasswordLastSet = 2026-05-11T07:14:00Z`, but Orchestrator's stored +credential record reports `PasswordLastSet = 2026-05-04T09:00:00Z` — +the platform is still sending the pre-rotation password. A sibling +unattended job (`OrderExporter` running as `UIPATH\USER2`) succeeded +on the same machine at `11:20Z`, so the host, RDP path, and network +are fine; the fault is scoped to the `USER1` credential. + +This maps to: +`references/products/orchestrator/playbooks/job-faulted-logon-failure.md` +(branch 3 — password mismatch / stale Orchestrator credential). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal background-unattended UiPath project (no UI activities — LogMessage + Delay only) | + +> **Note on fixtures.** Like sibling synthetic scenarios, the +> fixtures here were authored from the documented playbook +> signature rather than captured from a real `.investigation/` +> session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression +> signal. + +## How this differs from sibling cause-branches of the same playbook + +The `job-faulted-logon-failure` playbook covers six cause-branches. +The agent must select the right one — matching the **playbook** is +not enough. Distinguishing fingerprints: + +| Branch | Fingerprint distinguishing it from branch 3 | +|---|---| +| **1 — Session-config mismatch** | `RequiresUserInteraction: true` AND `LoginToConsole: false`. Our scenario has `false` / `true` respectively. | +| **2 — Account locked in AD** | LSA returns `STATUS_ACCOUNT_LOCKED_OUT` (`0xC0000234`); Windows code `0x00000775`; Robot log says `Account is locked`. Our log explicitly says `Account is NOT locked` and code is `0x0000052E`. | +| **3 — Password mismatch** *(this scenario)* | LSA returns `STATUS_LOGON_FAILURE` (`0xC000006D`) with sub-status `0xC000006A` (wrong password); Windows code `0x0000052E`; account NOT locked; AD `PasswordLastSet` recent AND Orchestrator `PasswordLastSet` stale. | +| **4 — Lockout loop (3 → 2 chained)** | Same as branch 3 PLUS account is currently locked AND `badPwdCount` climbing rapidly. Our account is not yet locked — agent should flag this as a *risk* (H5), not as the current state. | +| **5 — MFA / Conditional Access** | LSA / Entra returns interactive-auth-required; Robot log mentions MFA / Conditional Access. Absent in our fixtures. | +| **6 — RDP slot conflict** | Other users also fail on the same machine; `query session` would show a disconnected RDP session. Our sibling user `USER2` *succeeds* on the same machine → branch 6 ruled out. | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-diagnostics` skill +- Agent matched + `orchestrator/playbooks/job-faulted-logon-failure.md` AND + selected **branch 3 (password mismatch / stale Orchestrator + credential)** as the root cause +- Conclusion must explicitly name the stale Orchestrator + credential and propose the playbook's resolution sequence + (pause triggers → update stored credential → verify → unlock + if needed → re-enable triggers) +- Bonus (not required for pass): agent flags branch-4 lockout-loop + risk while triggers remain active + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name logon-failure-password-mismatch --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/09d516cd3a.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/09d516cd3a.json deleted file mode 100644 index 87594f7f71..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/09d516cd3a.json +++ /dev/null @@ -1,91 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent jobs in UnattendedDemo (broader history). RobotUser1 fails every attempt with the same 'Logon failed' error since 2026-05-12T10:55Z. RobotUser2 succeeded on the same machine (MOCK-HOST) at 2026-05-12T11:20Z — proves the machine itself is fine and isolates the problem to user RobotUser1's credential.", - "Data": [ - { - "Key": "aabb1122-3344-5566-7788-99aabbccdd00", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:30:00.412Z", - "EndTime": "2026-05-12T11:30:01.118Z", - "CreationTime": "2026-05-12T11:30:00.190Z" - }, - { - "Key": "bbcc2233-4455-6677-8899-aabbccddee22", - "State": "Successful", - "ProcessName": "OrderExporter", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:20:00.001Z", - "EndTime": "2026-05-12T11:20:12.880Z", - "CreationTime": "2026-05-12T11:19:59.812Z" - }, - { - "Key": "aabb1100-3344-5566-7788-99aabbccdd11", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:15:00.211Z", - "EndTime": "2026-05-12T11:15:00.987Z", - "CreationTime": "2026-05-12T11:14:59.980Z" - }, - { - "Key": "aabb1099-3344-5566-7788-99aabbccdd22", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:05:00.155Z", - "EndTime": "2026-05-12T11:05:00.901Z", - "CreationTime": "2026-05-12T11:04:59.880Z" - }, - { - "Key": "aabb1088-3344-5566-7788-99aabbccdd33", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T10:55:00.302Z", - "EndTime": "2026-05-12T10:55:01.040Z", - "CreationTime": "2026-05-12T10:54:59.991Z" - }, - { - "Key": "ddee4455-6677-8899-aabb-ccddee001122", - "State": "Successful", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T08:45:00.105Z", - "EndTime": "2026-05-12T08:45:07.211Z", - "CreationTime": "2026-05-12T08:44:59.821Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/1c1475ea15.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/1c1475ea15.json deleted file mode 100644 index 0b34b59894..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/1c1475ea15.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T11:30:01.110Z", - "Message": "Job 'aabb1122-3344-5566-7788-99aabbccdd00' failed to start. Could not start executor. Logon failed for user UIPATH\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T11:30:00.988Z", - "Message": "[Robot] Windows logon attempt for UIPATH\\USER1 on MOCK-HOST rejected by LSA. NTSTATUS=0xC000006D (STATUS_LOGON_FAILURE). Sub-status=0xC000006A (wrong password). Account is NOT locked. Caller: UiPath Robot service (unattended)." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T11:30:00.620Z", - "Message": "[Robot] Resolving credentials for UIPATH\\USER1 from Orchestrator unattended-robot record (LoginToConsole=True). Initiating Windows logon." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T11:30:00.205Z", - "Message": "Job 'aabb1122-3344-5566-7788-99aabbccdd00' created. Source=Manual, ReleaseName=InvoiceIngestor, RequiresUserInteraction=False, TargetUser=UIPATH\\USER1." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/4a7c6d2630.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/4a7c6d2630.json deleted file mode 100644 index eb20c79cc6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/4a7c6d2630.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "_doc": "Earlier failed run, same user (RobotUser1), same machine, identical Logon-failed error 0x0000052E. Establishes the persistent same-user pattern; an admin who only saw a single failure might dismiss it as transient, but three identical failures within 35 minutes rule out 'flaky'. The state 0x0000052E (bad password) — not 0x00000775 (locked) — is what distinguishes branch 3 from branches 2/4.", - "Data": { - "Key": "aabb1100-3344-5566-7788-99aabbccdd11", - "StartTime": "2026-05-12T11:15:00.211Z", - "EndTime": "2026-05-12T11:15:00.987Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "TimeTrigger", - "BatchExecutionKey": "9e1f8b22-2b3c-4d5e-9f0a-1b2c3d4e5f6b", - "Info": "Could not start executor. Logon failed for user UIPATH\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-12T11:14:59.980Z", - "ReleaseName": "InvoiceIngestor", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 4670077, - "OrganizationUnitId": 7791617, - "FolderKey": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T11:15:00.987Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "aabb1100-3344-5566-7788-99aabbccdd11", - "Id": 670911100 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/614ae8d27d.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/614ae8d27d.json deleted file mode 100644 index f3e6b664d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/614ae8d27d.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent Faulted jobs in UnattendedDemo. Three failures on RobotUser1 with identical 'Logon failed' error (0x0000052E) within ~25 minutes — same-user persistent failure pattern, classic branch 3 fingerprint.", - "Data": [ - { - "Key": "aabb1122-3344-5566-7788-99aabbccdd00", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:30:00.412Z", - "EndTime": "2026-05-12T11:30:01.118Z", - "CreationTime": "2026-05-12T11:30:00.190Z" - }, - { - "Key": "aabb1100-3344-5566-7788-99aabbccdd11", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:15:00.211Z", - "EndTime": "2026-05-12T11:15:00.987Z", - "CreationTime": "2026-05-12T11:14:59.980Z" - }, - { - "Key": "aabb1099-3344-5566-7788-99aabbccdd22", - "State": "Faulted", - "ProcessName": "InvoiceIngestor", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T11:05:00.155Z", - "EndTime": "2026-05-12T11:05:00.901Z", - "CreationTime": "2026-05-12T11:04:59.880Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/6db4592f78.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/6db4592f78.json deleted file mode 100644 index 1957c41473..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/6db4592f78.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Name": "UnattendedDemo", - "Path": "UnattendedDemo", - "Description": "Folder hosting the InvoiceIngestor background unattended process.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/6e219c488f.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/6e219c488f.json deleted file mode 100644 index 1905406ff1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/6e219c488f.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T11:30:01.110Z", - "Message": "Job 'aabb1122-3344-5566-7788-99aabbccdd00' failed to start. Could not start executor. Logon failed for user UIPATH\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T11:30:00.988Z", - "Message": "[Robot] Windows logon attempt for UIPATH\\USER1 on MOCK-HOST rejected by LSA. NTSTATUS=0xC000006D (STATUS_LOGON_FAILURE). Sub-status=0xC000006A (wrong password). Account is NOT locked. Caller: UiPath Robot service (unattended)." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/README.md deleted file mode 100644 index 36975cb2ef..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/README.md +++ /dev/null @@ -1,87 +0,0 @@ -# Logon Failure — Password Mismatch (Stale Orchestrator Credential) - -This scenario reproduces the **"Could Not Start Executor — Logon -Failed for User"** playbook's **branch 3**: the AD password was -rotated for the unattended robot user, but the credential stored in -Orchestrator was not updated to match. Every job since the rotation -faults with: - -``` -Could not start executor. Logon failed for user UIPATH\USER1. -The user name or password is incorrect. (0x0000052E). -Last error: 131092. RDP connection failed. -``` - -## What this scenario uncovers - -**Root Cause:** Three identical `InvoiceIngestor` runs (keys -`aabb1099-...`, `aabb1100-...`, `aabb1122-...`) faulted within -~25 minutes for service account `UIPATH\USER1` on `MOCK-HOST`. AD -`PasswordLastSet = 2026-05-11T07:14:00Z`, but Orchestrator's stored -credential record reports `PasswordLastSet = 2026-05-04T09:00:00Z` — -the platform is still sending the pre-rotation password. A sibling -unattended job (`OrderExporter` running as `UIPATH\USER2`) succeeded -on the same machine at `11:20Z`, so the host, RDP path, and network -are fine; the fault is scoped to the `USER1` credential. - -This maps to: -`references/products/orchestrator/playbooks/job-faulted-logon-failure.md` -(branch 3 — password mismatch / stale Orchestrator credential). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | minimal background-unattended UiPath project (no UI activities — LogMessage + Delay only) | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like sibling synthetic scenarios, the -> fixtures here were authored from the documented playbook -> signature rather than captured from a real `.investigation/` -> session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression -> signal. - -## How this differs from sibling cause-branches of the same playbook - -The `job-faulted-logon-failure` playbook covers six cause-branches. -The agent must select the right one — matching the **playbook** is -not enough. Distinguishing fingerprints: - -| Branch | Fingerprint distinguishing it from branch 3 | -|---|---| -| **1 — Session-config mismatch** | `RequiresUserInteraction: true` AND `LoginToConsole: false`. Our scenario has `false` / `true` respectively. | -| **2 — Account locked in AD** | LSA returns `STATUS_ACCOUNT_LOCKED_OUT` (`0xC0000234`); Windows code `0x00000775`; Robot log says `Account is locked`. Our log explicitly says `Account is NOT locked` and code is `0x0000052E`. | -| **3 — Password mismatch** *(this scenario)* | LSA returns `STATUS_LOGON_FAILURE` (`0xC000006D`) with sub-status `0xC000006A` (wrong password); Windows code `0x0000052E`; account NOT locked; AD `PasswordLastSet` recent AND Orchestrator `PasswordLastSet` stale. | -| **4 — Lockout loop (3 → 2 chained)** | Same as branch 3 PLUS account is currently locked AND `badPwdCount` climbing rapidly. Our account is not yet locked — agent should flag this as a *risk* (H5), not as the current state. | -| **5 — MFA / Conditional Access** | LSA / Entra returns interactive-auth-required; Robot log mentions MFA / Conditional Access. Absent in our fixtures. | -| **6 — RDP slot conflict** | Other users also fail on the same machine; `query session` would show a disconnected RDP session. Our sibling user `USER2` *succeeds* on the same machine → branch 6 ruled out. | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-diagnostics` skill -- Agent matched - `orchestrator/playbooks/job-faulted-logon-failure.md` AND - selected **branch 3 (password mismatch / stale Orchestrator - credential)** as the root cause -- Conclusion must explicitly name the stale Orchestrator - credential and propose the playbook's resolution sequence - (pause triggers → update stored credential → verify → unlock - if needed → re-enable triggers) -- Bonus (not required for pass): agent flags branch-4 lockout-loop - risk while triggers remain active - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name logon-failure-password-mismatch --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/c734b36651.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/c734b36651.json deleted file mode 100644 index 89f4fa693c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/c734b36651.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "_doc": "Failing job. The Info field carries the verbatim 'Could not start executor. Logon failed for user UIPATH\\\\USER1. (0x0000052E)' signature — the entry point into the job-faulted-logon-failure playbook. 0x0000052E is the Windows logon-failure code for 'unknown user name or bad password' — distinct from 0x00000775 (account locked) and 0x00000532 (password expired). Combined with the same-user persistent pattern in the broader jobs list and the AD/Orchestrator credential mismatch evidence below, this points at branch 3 (password mismatch / stale Orchestrator credential).", - "Data": { - "Key": "aabb1122-3344-5566-7788-99aabbccdd00", - "StartTime": "2026-05-12T11:30:00.412Z", - "EndTime": "2026-05-12T11:30:01.118Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9e1f8b22-2b3c-4d5e-9f0a-1b2c3d4e5f6a", - "Info": "Could not start executor. Logon failed for user UIPATH\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-12T11:30:00.190Z", - "ReleaseName": "InvoiceIngestor", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 4670077, - "OrganizationUnitId": 7791617, - "FolderKey": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T11:30:01.118Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "aabb1122-3344-5566-7788-99aabbccdd00", - "Id": 670911211 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/c77be39749.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/c77be39749.json deleted file mode 100644 index 07558d4ac1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/c77be39749.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "Result": "Success", - "Code": "UserGet", - "_doc": "RobotUser1 tenant user record. Shows LoginToConsole=true (rules out branch 1) AND surfaces the stored-credential metadata: PasswordLastSet is 2026-05-04T09:00:00Z (eight days ago) while the AD password was rotated 2026-05-11T07:14:00Z (yesterday) — the Orchestrator credential is stale by ~7 days. This is the core evidence for playbook branch 3 (password mismatch / stale credential).", - "Data": { - "Key": "c1d2e3f4-a5b6-c7d8-e9f0-1a2b3c4d5e6f", - "UserName": "user1", - "EmailAddress": "user1@example.invalid", - "DomainName": "UIPATH", - "FullName": "Service Account 1", - "Type": "User", - "UnattendedRobot": { - "ExecutionSettings": { - "LoginToConsole": true, - "RunOnlyOneJobAtATime": false, - "ResolutionWidth": 1920, - "ResolutionHeight": 1080, - "ResolutionDepth": 32, - "FontSmoothing": false - }, - "UserName": "UIPATH\\USER1", - "CredentialStoreId": null, - "CredentialType": "Default", - "PasswordLastSet": "2026-05-04T09:00:00Z" - }, - "TenantId": 1, - "Id": 7501, - "_meta": { - "ad_password_last_set_observed": "2026-05-11T07:14:00Z", - "ad_password_last_set_source": "user-supplied in initial report; verifiable via 'Get-ADUser user1 -Properties PasswordLastSet' on a domain controller", - "comment": "Orchestrator-stored PasswordLastSet (2026-05-04) is older than the AD password rotation (2026-05-11). Stale credential — branch 3." - } - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/fce2357a00.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/fce2357a00.json deleted file mode 100644 index 2dd9d9ba3a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/fce2357a00.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "_doc": "Successful run on the SAME machine (MOCK-HOST) at 11:20Z — but for RobotUser2 (the OrderExporter process). Demonstrates: the machine itself is healthy, the network path to the host is fine, RDP works for some users. Isolates the failure to RobotUser1's credential specifically — rules out branches 1 (session-config — would apply to both users) and 6 (RDP slot conflict — would block USER2 too if a human were holding the slot).", - "Data": { - "Key": "bbcc2233-4455-6677-8899-aabbccddee22", - "StartTime": "2026-05-12T11:20:00.001Z", - "EndTime": "2026-05-12T11:20:12.880Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "TimeTrigger", - "BatchExecutionKey": "8c1d6b30-09a2-4b5c-8e7f-1a2b3c4d5e6c", - "Info": "", - "CreationTime": "2026-05-12T11:19:59.812Z", - "ReleaseName": "OrderExporter", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 4670099, - "OrganizationUnitId": 7791617, - "FolderKey": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER2", - "OrchestratorUserIdentity": "user2", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T11:20:12.880Z", - "JobError": null, - "ErrorCode": null, - "TraceId": "bbcc2233-4455-6677-8899-aabbccddee22", - "Id": 670911180 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/manifest.json deleted file mode 100644 index 535c2c3665..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for logon-failure-password-mismatch scenario (job-faulted-logon-failure playbook, branch 3 — stale Orchestrator credential after AD password rotation). Folder/job/user keys are stable; the agent derives them by chaining: folders list-current-user -> jobs list --state Faulted -> jobs get -> jobs logs -> robots list -> users get. The same-user-different-machine and same-machine-different-user evidence is surfaced via the broader jobs-list call (without --state filter).", - "rules": [ - { - "match": "or folders list --output json", - "file": "6db4592f78.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --state Faulted --output json", - "file": "614ae8d27d.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --state Faulted --output json", - "file": "614ae8d27d.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --output json", - "file": "09d516cd3a.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --output json", - "file": "09d516cd3a.json" - }, - { - "match": "or jobs get \"aabb1122-3344-5566-7788-99aabbccdd00\" --output json", - "file": "c734b36651.json" - }, - { - "match": "or jobs get aabb1122-3344-5566-7788-99aabbccdd00 --output json", - "file": "c734b36651.json" - }, - { - "match": "or jobs get \"aabb1100-3344-5566-7788-99aabbccdd11\" --output json", - "file": "4a7c6d2630.json" - }, - { - "match": "or jobs get aabb1100-3344-5566-7788-99aabbccdd11 --output json", - "file": "4a7c6d2630.json" - }, - { - "match": "or jobs get \"bbcc2233-4455-6677-8899-aabbccddee22\" --output json", - "file": "fce2357a00.json" - }, - { - "match": "or jobs get bbcc2233-4455-6677-8899-aabbccddee22 --output json", - "file": "fce2357a00.json" - }, - { - "match": "or jobs logs \"aabb1122-3344-5566-7788-99aabbccdd00\" --level Error --output json", - "file": "6e219c488f.json" - }, - { - "match": "or jobs logs aabb1122-3344-5566-7788-99aabbccdd00 --level Error --output json", - "file": "6e219c488f.json" - }, - { - "match": "or jobs logs \"aabb1122-3344-5566-7788-99aabbccdd00\" --output json", - "file": "1c1475ea15.json" - }, - { - "match": "or jobs logs aabb1122-3344-5566-7788-99aabbccdd00 --output json", - "file": "1c1475ea15.json" - }, - { - "match": "or users get \"c1d2e3f4-a5b6-c7d8-e9f0-1a2b3c4d5e6f\" --output json", - "file": "c77be39749.json" - }, - { - "match": "or users get c1d2e3f4-a5b6-c7d8-e9f0-1a2b3c4d5e6f --output json", - "file": "c77be39749.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..d88e1cfdbe --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/data/uip-fixture.json @@ -0,0 +1,131 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"aabb1122-3344-5566-7788-99aabbccdd00\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:30:00.412Z\",\n \"EndTime\": \"2026-05-12T11:30:01.118Z\",\n \"CreationTime\": \"2026-05-12T11:30:00.190Z\"\n },\n {\n \"Key\": \"aabb1100-3344-5566-7788-99aabbccdd11\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:15:00.211Z\",\n \"EndTime\": \"2026-05-12T11:15:00.987Z\",\n \"CreationTime\": \"2026-05-12T11:14:59.980Z\"\n },\n {\n \"Key\": \"aabb1099-3344-5566-7788-99aabbccdd22\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:05:00.155Z\",\n \"EndTime\": \"2026-05-12T11:05:00.901Z\",\n \"CreationTime\": \"2026-05-12T11:04:59.880Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aabb1122-3344-5566-7788-99aabbccdd00", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T11:30:01.110Z\",\r\n \"Message\": \"Job 'aabb1122-3344-5566-7788-99aabbccdd00' failed to start. Could not start executor. Logon failed for user UIPATH\\\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T11:30:00.988Z\",\r\n \"Message\": \"[Robot] Windows logon attempt for UIPATH\\\\USER1 on MOCK-HOST rejected by LSA. NTSTATUS=0xC000006D (STATUS_LOGON_FAILURE). Sub-status=0xC000006A (wrong password). Account is NOT locked. Caller: UiPath Robot service (unattended).\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"aabb1122-3344-5566-7788-99aabbccdd00\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:30:00.412Z\",\n \"EndTime\": \"2026-05-12T11:30:01.118Z\",\n \"CreationTime\": \"2026-05-12T11:30:00.190Z\"\n },\n {\n \"Key\": \"bbcc2233-4455-6677-8899-aabbccddee22\",\n \"State\": \"Successful\",\n \"ProcessName\": \"OrderExporter\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:20:00.001Z\",\n \"EndTime\": \"2026-05-12T11:20:12.880Z\",\n \"CreationTime\": \"2026-05-12T11:19:59.812Z\"\n },\n {\n \"Key\": \"aabb1100-3344-5566-7788-99aabbccdd11\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:15:00.211Z\",\n \"EndTime\": \"2026-05-12T11:15:00.987Z\",\n \"CreationTime\": \"2026-05-12T11:14:59.980Z\"\n },\n {\n \"Key\": \"aabb1099-3344-5566-7788-99aabbccdd22\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T11:05:00.155Z\",\n \"EndTime\": \"2026-05-12T11:05:00.901Z\",\n \"CreationTime\": \"2026-05-12T11:04:59.880Z\"\n },\n {\n \"Key\": \"aabb1088-3344-5566-7788-99aabbccdd33\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T10:55:00.302Z\",\n \"EndTime\": \"2026-05-12T10:55:01.040Z\",\n \"CreationTime\": \"2026-05-12T10:54:59.991Z\"\n },\n {\n \"Key\": \"ddee4455-6677-8899-aabb-ccddee001122\",\n \"State\": \"Successful\",\n \"ProcessName\": \"InvoiceIngestor\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T08:45:00.105Z\",\n \"EndTime\": \"2026-05-12T08:45:07.211Z\",\n \"CreationTime\": \"2026-05-12T08:44:59.821Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aabb1122-3344-5566-7788-99aabbccdd00", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobGet\",\n \"Data\": {\n \"Key\": \"aabb1122-3344-5566-7788-99aabbccdd00\",\n \"StartTime\": \"2026-05-12T11:30:00.412Z\",\n \"EndTime\": \"2026-05-12T11:30:01.118Z\",\n \"State\": \"Faulted\",\n \"JobPriority\": \"Normal\",\n \"SpecificPriorityValue\": 50,\n \"Source\": \"Manual\",\n \"SourceType\": \"Manual\",\n \"BatchExecutionKey\": \"9e1f8b22-2b3c-4d5e-9f0a-1b2c3d4e5f6a\",\n \"Info\": \"Could not start executor. Logon failed for user UIPATH\\\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\n \"CreationTime\": \"2026-05-12T11:30:00.190Z\",\n \"ReleaseName\": \"InvoiceIngestor\",\n \"Type\": \"Unattended\",\n \"EnvironmentVariables\": \"\",\n \"OutputArguments\": \"{}\",\n \"HostMachineName\": \"MOCK-HOST\",\n \"HasMediaRecorded\": false,\n \"HasVideoRecorded\": false,\n \"RuntimeType\": \"Unattended\",\n \"RequiresUserInteraction\": false,\n \"ReleaseVersionId\": 4670077,\n \"OrganizationUnitId\": 7791617,\n \"FolderKey\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\n \"Reference\": \"\",\n \"ProcessType\": \"Process\",\n \"ResumeOnSameContext\": false,\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\n \"OrchestratorUserIdentity\": \"user1\",\n \"RemoteControlAccess\": \"None\",\n \"LastModificationTime\": \"2026-05-12T11:30:01.118Z\",\n \"JobError\": {},\n \"ErrorCode\": \"Robot\",\n \"TraceId\": \"aabb1122-3344-5566-7788-99aabbccdd00\",\n \"Id\": 670911211\n }\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aabb1100-3344-5566-7788-99aabbccdd11", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobGet\",\n \"Data\": {\n \"Key\": \"aabb1100-3344-5566-7788-99aabbccdd11\",\n \"StartTime\": \"2026-05-12T11:15:00.211Z\",\n \"EndTime\": \"2026-05-12T11:15:00.987Z\",\n \"State\": \"Faulted\",\n \"JobPriority\": \"Normal\",\n \"SpecificPriorityValue\": 50,\n \"Source\": \"Schedule\",\n \"SourceType\": \"TimeTrigger\",\n \"BatchExecutionKey\": \"9e1f8b22-2b3c-4d5e-9f0a-1b2c3d4e5f6b\",\n \"Info\": \"Could not start executor. Logon failed for user UIPATH\\\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\n \"CreationTime\": \"2026-05-12T11:14:59.980Z\",\n \"ReleaseName\": \"InvoiceIngestor\",\n \"Type\": \"Unattended\",\n \"EnvironmentVariables\": \"\",\n \"OutputArguments\": \"{}\",\n \"HostMachineName\": \"MOCK-HOST\",\n \"HasMediaRecorded\": false,\n \"HasVideoRecorded\": false,\n \"RuntimeType\": \"Unattended\",\n \"RequiresUserInteraction\": false,\n \"ReleaseVersionId\": 4670077,\n \"OrganizationUnitId\": 7791617,\n \"FolderKey\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\n \"Reference\": \"\",\n \"ProcessType\": \"Process\",\n \"ResumeOnSameContext\": false,\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\n \"OrchestratorUserIdentity\": \"user1\",\n \"RemoteControlAccess\": \"None\",\n \"LastModificationTime\": \"2026-05-12T11:15:00.987Z\",\n \"JobError\": {},\n \"ErrorCode\": \"Robot\",\n \"TraceId\": \"aabb1100-3344-5566-7788-99aabbccdd11\",\n \"Id\": 670911100\n }\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bbcc2233-4455-6677-8899-aabbccddee22", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobGet\",\n \"Data\": {\n \"Key\": \"bbcc2233-4455-6677-8899-aabbccddee22\",\n \"StartTime\": \"2026-05-12T11:20:00.001Z\",\n \"EndTime\": \"2026-05-12T11:20:12.880Z\",\n \"State\": \"Successful\",\n \"JobPriority\": \"Normal\",\n \"SpecificPriorityValue\": 50,\n \"Source\": \"Schedule\",\n \"SourceType\": \"TimeTrigger\",\n \"BatchExecutionKey\": \"8c1d6b30-09a2-4b5c-8e7f-1a2b3c4d5e6c\",\n \"Info\": \"\",\n \"CreationTime\": \"2026-05-12T11:19:59.812Z\",\n \"ReleaseName\": \"OrderExporter\",\n \"Type\": \"Unattended\",\n \"EnvironmentVariables\": \"\",\n \"OutputArguments\": \"{}\",\n \"HostMachineName\": \"MOCK-HOST\",\n \"HasMediaRecorded\": false,\n \"HasVideoRecorded\": false,\n \"RuntimeType\": \"Unattended\",\n \"RequiresUserInteraction\": false,\n \"ReleaseVersionId\": 4670099,\n \"OrganizationUnitId\": 7791617,\n \"FolderKey\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\n \"Reference\": \"\",\n \"ProcessType\": \"Process\",\n \"ResumeOnSameContext\": false,\n \"LocalSystemAccount\": \"UIPATH\\\\USER2\",\n \"OrchestratorUserIdentity\": \"user2\",\n \"RemoteControlAccess\": \"None\",\n \"LastModificationTime\": \"2026-05-12T11:20:12.880Z\",\n \"JobError\": null,\n \"ErrorCode\": null,\n \"TraceId\": \"bbcc2233-4455-6677-8899-aabbccddee22\",\n \"Id\": 670911180\n }\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aabb1122-3344-5566-7788-99aabbccdd00", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T11:30:01.110Z\",\r\n \"Message\": \"Job 'aabb1122-3344-5566-7788-99aabbccdd00' failed to start. Could not start executor. Logon failed for user UIPATH\\\\USER1. The user name or password is incorrect. (0x0000052E). Last error: 131092. RDP connection failed.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T11:30:00.988Z\",\r\n \"Message\": \"[Robot] Windows logon attempt for UIPATH\\\\USER1 on MOCK-HOST rejected by LSA. NTSTATUS=0xC000006D (STATUS_LOGON_FAILURE). Sub-status=0xC000006A (wrong password). Account is NOT locked. Caller: UiPath Robot service (unattended).\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T11:30:00.620Z\",\r\n \"Message\": \"[Robot] Resolving credentials for UIPATH\\\\USER1 from Orchestrator unattended-robot record (LoginToConsole=True). Initiating Windows logon.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T11:30:00.205Z\",\r\n \"Message\": \"Job 'aabb1122-3344-5566-7788-99aabbccdd00' created. Source=Manual, ReleaseName=InvoiceIngestor, RequiresUserInteraction=False, TargetUser=UIPATH\\\\USER1.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "get", + "c1d2e3f4-a5b6-c7d8-e9f0-1a2b3c4d5e6f", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"UserGet\",\n \"Data\": {\n \"Key\": \"c1d2e3f4-a5b6-c7d8-e9f0-1a2b3c4d5e6f\",\n \"UserName\": \"user1\",\n \"EmailAddress\": \"user1@example.invalid\",\n \"DomainName\": \"UIPATH\",\n \"FullName\": \"Service Account 1\",\n \"Type\": \"User\",\n \"UnattendedRobot\": {\n \"ExecutionSettings\": {\n \"LoginToConsole\": true,\n \"RunOnlyOneJobAtATime\": false,\n \"ResolutionWidth\": 1920,\n \"ResolutionHeight\": 1080,\n \"ResolutionDepth\": 32,\n \"FontSmoothing\": false\n },\n \"UserName\": \"UIPATH\\\\USER1\",\n \"CredentialStoreId\": null,\n \"CredentialType\": \"Default\",\n \"PasswordLastSet\": \"2026-05-04T09:00:00Z\"\n },\n \"TenantId\": 1,\n \"Id\": 7501\n }\n}\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Name\": \"UnattendedDemo\",\r\n \"Path\": \"UnattendedDemo\",\r\n \"Description\": \"Folder hosting the InvoiceIngestor background unattended process.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/task.yaml index 7b58e1badc..cf0afb9654 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/logon-failure-password-mismatch/task.yaml @@ -29,21 +29,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/README.md new file mode 100644 index 0000000000..923aad27f2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/README.md @@ -0,0 +1,59 @@ +# No Host Pending — Genuine No-Available-Host + +This scenario replays a UiPath troubleshooting investigation where a job is +stuck in `Pending` because **no host is connected to the folder's machine +template** — a genuine no-available-host condition. + +## What the scenario covers + +A user reports their last job in the Shared folder is stuck in `Pending` and +never started. The agent discovers exactly one Pending job there +(process `NewBlankTask`, Unattended) whose `PendingReasons.ErrorCodes` is the +host-family (`TemplateNoHostsAvailable`, `DynamicJobConnectedMachinesInvalid`, +`DynamicJobConnectedMachinesWindowsRobotVersionInvalid`) with the error text +*"…there is none connected to this folder"*. + +The evidence confirms a real no-host condition: + +- `machines list --all-fields`: the only Default-scope template **DanLaptopNew** + has `robotVersions: []` — **no runtime is connected to it**. (The other + templates are Serverless / PersonalWorkspace scope, not eligible for this + Default-scope folder job.) +- `licenses info`: Unattended `Allowed=1`, `Used=0` — the slot is **free**, so + license is not the blocker. +- `JobHistory`: a single Pending entry — the job never started because nothing + could accept it. + +So no machine is available to run the job. The fix is to **connect/assign a +machine to the folder's template** (sign a robot into a machine on DanLaptopNew, +or assign a template that has a connected host). Re-triggering does **not** help +while no host is connected — a fresh dispatch finds the same empty template. + +This is the *paired counterpart* to `no-login-pending/` (a +`RobotNoMatchingUsernames` credentials case). Together the two tests verify the +skill routes correctly between `job-pending-no-host.md`, `robot-credentials.md`, +and `job-pending-stale-dispatch.md` based on the actual evidence (here: no +connected runtime), not on superficial wording. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +(No `process/` snapshot — this investigation is purely CLI-driven; no project +source was read.) + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- The agent's presented resolution names the **no available host** (no runtime + connected to the folder's template) as the root cause and recommends + **connecting/assigning a machine to the folder**, without falling for the + stale-dispatch/re-trigger, license-exhaustion, or credential-mismatch + misreads. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/0e63a71fda.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/0e63a71fda.json deleted file mode 100644 index 7b6d64075b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/0e63a71fda.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Pending", - "Timestamp": "2026-05-28T08:48:18.620Z", - "JobId": 65401221 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/2dced5ed3a.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/2dced5ed3a.json deleted file mode 100644 index 22418ac8be..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/2dced5ed3a.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "cb00355c-3835-4c18-b37e-af5c7609c599", - "State": "Pending", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "711bbba5-c21b-45c0-a556-c18ef5f8b516", - "CreationTime": "2026-05-28T08:48:18.580Z", - "ReleaseName": "NewBlankTask", - "Type": "Unattended", - "EnvironmentVariables": "", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "", - "RemoteControlAccess": "None", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 65401221, - "PendingReasons": { - "IsValid": false, - "Errors": [ - "Windows process. This job can only run on a windows machine and there is none connected to this folder.", - "Windows .NET5 process. This job can only run on a windows machine with a minimum 21.10 robot version and there is none connected to this folder." - ], - "ErrorCodes": [ - "TemplateNoHostsAvailable", - "DynamicJobConnectedMachinesInvalid", - "DynamicJobConnectedMachinesWindowsRobotVersionInvalid" - ] - } - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/684aa172ef.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/684aa172ef.json deleted file mode 100644 index f542402a36..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/684aa172ef.json +++ /dev/null @@ -1,90 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "licenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2062", - "name": "DanLaptopNew", - "description": "", - "type": "Template", - "scope": "Default", - "nonProductionSlots": 0, - "unattendedSlots": 1, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "b8d82bd0-1d19-4a23-8b34-fc26927d4cfa", - "endpointDetectionStatus": "NotAvailable", - "robotVersions": [], - "robotUsers": [], - "automationType": "Any", - "targetFramework": "Any", - "tags": [], - "id": 1890824 - }, - { - "licenseKey": "78d93cb4-23ae-4d30-9f56-2640d6584221", - "name": "[Default] Cloud Robots - Serverless", - "description": "Automatically generated for use in Personal Workspaces", - "type": "Template", - "scope": "Serverless", - "nonProductionSlots": 0, - "unattendedSlots": 0, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "3b82fc7a-8154-4c17-ac05-60f026904b8b", - "endpointDetectionStatus": "NotAvailable", - "robotVersions": [], - "robotUsers": [], - "automationType": "Any", - "targetFramework": "Portable", - "tags": [], - "id": 1104780 - }, - { - "name": "original_email@test.com's workspace machine", - "type": "Template", - "scope": "PersonalWorkspace", - "nonProductionSlots": 0, - "unattendedSlots": 1, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "6f12f3e7-75f5-475e-aa34-3f5bd5c564e1", - "endpointDetectionStatus": "NotAvailable", - "robotVersions": [ - { - "count": 1, - "version": "26.0.193-cloud.23059" - } - ], - "robotUsers": [], - "automationType": "Any", - "targetFramework": "Any", - "tags": [], - "id": 961460 - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/6b61cd95e8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/6b61cd95e8.json deleted file mode 100644 index 9a474a525f..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/6b61cd95e8.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "LicenseInfo", - "Data": { - "IsExpired": false, - "SubscriptionPlan": "ENTERPRISE", - "Allowed": { - "Attended": 0, - "PerformanceTest": 0, - "Headless": 0, - "NonProduction": 0, - "BusinessRule": 0, - "ProcessOrchestration": 0, - "AgentService": 0, - "StudioX": 0, - "CaseManagement": 0, - "Local": 0, - "Unattended": 1, - "Development": 0, - "Flow": 0, - "TestAutomation": 1, - "AppTest": 0, - "Hosting": 0, - "StudioPro": 0 - }, - "Used": { - "Attended": 0, - "PerformanceTest": 0, - "Headless": 0, - "NonProduction": 0, - "BusinessRule": 0, - "ProcessOrchestration": 0, - "AgentService": 0, - "StudioX": 0, - "CaseManagement": 0, - "Local": 0, - "Unattended": 0, - "Development": 0, - "Flow": 0, - "TestAutomation": 0, - "AppTest": 0, - "Hosting": 0, - "StudioPro": 0 - }, - "LicensedFeatures": [] - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/6d75a2ad66.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/6d75a2ad66.json deleted file mode 100644 index 2a3b4373a3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/6d75a2ad66.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/97259002c9.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/97259002c9.json deleted file mode 100644 index ce4dbdc1fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/97259002c9.json +++ /dev/null @@ -1,349 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "cb00355c-3835-4c18-b37e-af5c7609c599", - "State": "Pending", - "ProcessName": "NewBlankTask", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-28T08:48:18.580Z" - }, - { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-28T08:00:50.870Z", - "EndTime": "2026-05-28T08:01:02.417Z", - "CreationTime": "2026-05-27T16:24:24.603Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-26T08:48:20.617Z", - "EndTime": "2026-05-26T08:48:20.617Z", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 24, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} - diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/README.md deleted file mode 100644 index a27270a789..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/README.md +++ /dev/null @@ -1,58 +0,0 @@ -# No Host Pending — Genuine No-Available-Host - -This scenario replays a UiPath troubleshooting investigation where a job is -stuck in `Pending` because **no host is connected to the folder's machine -template** — a genuine no-available-host condition. - -## What the scenario covers - -A user reports their last job in the Shared folder is stuck in `Pending` and -never started. The agent discovers exactly one Pending job there -(process `NewBlankTask`, Unattended) whose `PendingReasons.ErrorCodes` is the -host-family (`TemplateNoHostsAvailable`, `DynamicJobConnectedMachinesInvalid`, -`DynamicJobConnectedMachinesWindowsRobotVersionInvalid`) with the error text -*"…there is none connected to this folder"*. - -The evidence confirms a real no-host condition: - -- `machines list --all-fields`: the only Default-scope template **DanLaptopNew** - has `robotVersions: []` — **no runtime is connected to it**. (The other - templates are Serverless / PersonalWorkspace scope, not eligible for this - Default-scope folder job.) -- `licenses info`: Unattended `Allowed=1`, `Used=0` — the slot is **free**, so - license is not the blocker. -- `JobHistory`: a single Pending entry — the job never started because nothing - could accept it. - -So no machine is available to run the job. The fix is to **connect/assign a -machine to the folder's template** (sign a robot into a machine on DanLaptopNew, -or assign a template that has a connected host). Re-triggering does **not** help -while no host is connected — a fresh dispatch finds the same empty template. - -This is the *paired counterpart* to `no-login-pending/` (a -`RobotNoMatchingUsernames` credentials case). Together the two tests verify the -skill routes correctly between `job-pending-no-host.md`, `robot-credentials.md`, -and `job-pending-stale-dispatch.md` based on the actual evidence (here: no -connected runtime), not on superficial wording. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | recorded `uip` CLI responses for the scenario | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -(No `process/` snapshot — this investigation is purely CLI-driven; no project -source was read.) - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- The agent's presented resolution names the **no available host** (no runtime - connected to the folder's template) as the root cause and recommends - **connecting/assigning a machine to the folder**, without falling for the - stale-dispatch/re-trigger, license-exhaustion, or credential-mismatch - misreads. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/fda63e3155.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/fda63e3155.json deleted file mode 100644 index a32f48ae02..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/fda63e3155.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "Name": "DanLaptopNew", - "ID": 1890824, - "Key": "b8d82bd0-1d19-4a23-8b34-fc26927d4cfa", - "Type": "Template", - "Scope": "Default", - "Description": "", - "UnattendedSlots": 1, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2062" - }, - { - "Name": "[Default] Cloud Robots - Serverless", - "ID": 1104780, - "Key": "3b82fc7a-8154-4c17-ac05-60f026904b8b", - "Type": "Template", - "Scope": "Serverless", - "Description": "Automatically generated for use in Personal Workspaces", - "UnattendedSlots": 0, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "78d93cb4-23ae-4d30-9f56-2640d6584221" - }, - { - "Name": "original_email@test.com's workspace machine", - "ID": 961460, - "Key": "6f12f3e7-75f5-475e-aa34-3f5bd5c564e1", - "Type": "Template", - "Scope": "PersonalWorkspace", - "Description": "", - "UnattendedSlots": 1, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/manifest.json deleted file mode 100644 index 909f2cbe95..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/m/r/manifest.json +++ /dev/null @@ -1,43 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --state Running --output json", - "file": "6d75a2ad66.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1 --output json", - "file": "97259002c9.json" - }, - { - "match": "or jobs get cb00355c-3835-4c18-b37e-af5c7609c599 --output json", - "file": "2dced5ed3a.json" - }, - { - "match": "or jobs history cb00355c-3835-4c18-b37e-af5c7609c599 --output json", - "file": "0e63a71fda.json" - }, - { - "match": "or machines list --all-fields --output json", - "file": "684aa172ef.json" - }, - { - "match": "or machines list --output json", - "file": "fda63e3155.json" - }, - { - "match": "or licenses info --output json", - "file": "6b61cd95e8.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/uip-fixture.json new file mode 100644 index 0000000000..d476b550ea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/data/uip-fixture.json @@ -0,0 +1,114 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--state", + "Running", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"cb00355c-3835-4c18-b37e-af5c7609c599\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"NewBlankTask\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-28T08:48:18.580Z\"\r\n },\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-28T08:00:50.870Z\",\r\n \"EndTime\": \"2026-05-28T08:01:02.417Z\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 24,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "cb00355c-3835-4c18-b37e-af5c7609c599", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"cb00355c-3835-4c18-b37e-af5c7609c599\",\r\n \"State\": \"Pending\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"711bbba5-c21b-45c0-a556-c18ef5f8b516\",\r\n \"CreationTime\": \"2026-05-28T08:48:18.580Z\",\r\n \"ReleaseName\": \"NewBlankTask\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 65401221,\r\n \"PendingReasons\": {\r\n \"IsValid\": false,\r\n \"Errors\": [\r\n \"Windows process. This job can only run on a windows machine and there is none connected to this folder.\",\r\n \"Windows .NET5 process. This job can only run on a windows machine with a minimum 21.10 robot version and there is none connected to this folder.\"\r\n ],\r\n \"ErrorCodes\": [\r\n \"TemplateNoHostsAvailable\",\r\n \"DynamicJobConnectedMachinesInvalid\",\r\n \"DynamicJobConnectedMachinesWindowsRobotVersionInvalid\"\r\n ]\r\n }\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "cb00355c-3835-4c18-b37e-af5c7609c599", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-05-28T08:48:18.620Z\",\r\n \"JobId\": 65401221\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--all-fields", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"licenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2062\",\r\n \"name\": \"DanLaptopNew\",\r\n \"description\": \"\",\r\n \"type\": \"Template\",\r\n \"scope\": \"Default\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 1,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"b8d82bd0-1d19-4a23-8b34-fc26927d4cfa\",\r\n \"endpointDetectionStatus\": \"NotAvailable\",\r\n \"robotVersions\": [],\r\n \"robotUsers\": [],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Any\",\r\n \"tags\": [],\r\n \"id\": 1890824\r\n },\r\n {\r\n \"licenseKey\": \"78d93cb4-23ae-4d30-9f56-2640d6584221\",\r\n \"name\": \"[Default] Cloud Robots - Serverless\",\r\n \"description\": \"Automatically generated for use in Personal Workspaces\",\r\n \"type\": \"Template\",\r\n \"scope\": \"Serverless\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 0,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"3b82fc7a-8154-4c17-ac05-60f026904b8b\",\r\n \"endpointDetectionStatus\": \"NotAvailable\",\r\n \"robotVersions\": [],\r\n \"robotUsers\": [],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Portable\",\r\n \"tags\": [],\r\n \"id\": 1104780\r\n },\r\n {\r\n \"name\": \"original_email@test.com's workspace machine\",\r\n \"type\": \"Template\",\r\n \"scope\": \"PersonalWorkspace\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 1,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"6f12f3e7-75f5-475e-aa34-3f5bd5c564e1\",\r\n \"endpointDetectionStatus\": \"NotAvailable\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"26.0.193-cloud.23059\"\r\n }\r\n ],\r\n \"robotUsers\": [],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Any\",\r\n \"tags\": [],\r\n \"id\": 961460\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"DanLaptopNew\",\r\n \"ID\": 1890824,\r\n \"Key\": \"b8d82bd0-1d19-4a23-8b34-fc26927d4cfa\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"Default\",\r\n \"Description\": \"\",\r\n \"UnattendedSlots\": 1,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2062\"\r\n },\r\n {\r\n \"Name\": \"[Default] Cloud Robots - Serverless\",\r\n \"ID\": 1104780,\r\n \"Key\": \"3b82fc7a-8154-4c17-ac05-60f026904b8b\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"Serverless\",\r\n \"Description\": \"Automatically generated for use in Personal Workspaces\",\r\n \"UnattendedSlots\": 0,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"78d93cb4-23ae-4d30-9f56-2640d6584221\"\r\n },\r\n {\r\n \"Name\": \"original_email@test.com's workspace machine\",\r\n \"ID\": 961460,\r\n \"Key\": \"6f12f3e7-75f5-475e-aa34-3f5bd5c564e1\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"PersonalWorkspace\",\r\n \"Description\": \"\",\r\n \"UnattendedSlots\": 1,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "licenses", + "info", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LicenseInfo\",\r\n \"Data\": {\r\n \"IsExpired\": false,\r\n \"SubscriptionPlan\": \"ENTERPRISE\",\r\n \"Allowed\": {\r\n \"Attended\": 0,\r\n \"PerformanceTest\": 0,\r\n \"Headless\": 0,\r\n \"NonProduction\": 0,\r\n \"BusinessRule\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"AgentService\": 0,\r\n \"StudioX\": 0,\r\n \"CaseManagement\": 0,\r\n \"Local\": 0,\r\n \"Unattended\": 1,\r\n \"Development\": 0,\r\n \"Flow\": 0,\r\n \"TestAutomation\": 1,\r\n \"AppTest\": 0,\r\n \"Hosting\": 0,\r\n \"StudioPro\": 0\r\n },\r\n \"Used\": {\r\n \"Attended\": 0,\r\n \"PerformanceTest\": 0,\r\n \"Headless\": 0,\r\n \"NonProduction\": 0,\r\n \"BusinessRule\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"AgentService\": 0,\r\n \"StudioX\": 0,\r\n \"CaseManagement\": 0,\r\n \"Local\": 0,\r\n \"Unattended\": 0,\r\n \"Development\": 0,\r\n \"Flow\": 0,\r\n \"TestAutomation\": 0,\r\n \"AppTest\": 0,\r\n \"Hosting\": 0,\r\n \"StudioPro\": 0\r\n },\r\n \"LicensedFeatures\": []\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/task.yaml index 53a486598c..f8c9aee85d 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-host-pending/task.yaml @@ -27,20 +27,10 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/README.md new file mode 100644 index 0000000000..fe541988e5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/README.md @@ -0,0 +1,48 @@ +# No Login Pending — Faithful Replay + +This scenario replays a real UiPath troubleshooting investigation where the +agent reached a verified resolution. The fixtures are the verbatim `uip` +CLI responses captured from that session. + +## What the original session uncovered + +A user reported their last job in the Shared folder was stuck in `Pending` +and never started. The agent discovered exactly one Pending job there +(`0fd7bea5-…`, process ERN, Unattended) whose `PendingReasons.ErrorCodes` +was `["RobotNoMatchingUsernames"]` — the robot account's credential-store +username did not match any machine-user mapping on the eligible machine +template (`DanLaptopNew`). The fix is to align the robot credentials with +the machine user, assign a matching account to the folder, or run the job +under a different account. + +## How this test reproduces it + +| Layer | Source | +|---|---| + +(No `process/` snapshot — this investigation is purely CLI-driven; no +project source was read.) + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- The agent's presented resolution names `RobotNoMatchingUsernames` / + credential-mismatch as the root cause and recommends one of the three + documented remediations in `robot-credentials.md`, without falling for + the no-host / stale-dispatch / license-exhaustion misreads + +## Re-running the extraction + +If the source transcript changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --transcript \ + --scenario-name no-login-pending --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/12363ea7c4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/12363ea7c4.json deleted file mode 100644 index d2f1521da2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/12363ea7c4.json +++ /dev/null @@ -1,334 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Pending", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "", - "EndTime": "", - "CreationTime": "2026-05-27T16:24:24.603Z" - }, - { - "Key": "6847a94d-68aa-4cfe-93dd-572a1c570185", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-27T16:19:15.317Z", - "EndTime": "2026-05-27T16:20:00.493Z", - "CreationTime": "2026-05-27T16:19:14.617Z" - }, - { - "Key": "3163478d-6d95-48cb-b447-a358af3813cc", - "State": "Faulted", - "ProcessName": "BlankProcess1", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-26T08:50:22.467Z", - "EndTime": "2026-05-26T08:50:57.280Z", - "CreationTime": "2026-05-26T08:50:21.833Z" - }, - { - "Key": "777970ae-5c98-454f-85a2-3325dcc98f81", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-26T08:48:20.617Z", - "EndTime": "2026-05-26T08:48:20.617Z", - "CreationTime": "2026-05-25T17:58:16.720Z" - }, - { - "Key": "66f6f28f-e41c-4660-aec4-374a4180ba28", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-25T17:57:45.707Z", - "EndTime": "2026-05-25T17:57:45.707Z", - "CreationTime": "2026-05-25T17:57:32.123Z" - }, - { - "Key": "8c994220-a19a-4c62-9868-6df7b61c97e3", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T17:56:44.133Z", - "EndTime": "2026-05-25T17:56:48.883Z", - "CreationTime": "2026-05-25T17:56:43.687Z" - }, - { - "Key": "c5907f72-8a22-410b-9d66-260a3a4ec46d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T12:34:25.143Z", - "EndTime": "2026-05-25T12:34:29.537Z", - "CreationTime": "2026-05-25T12:34:25.143Z" - }, - { - "Key": "91d543d2-1b71-4233-a94a-a1389014c998", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-25T10:38:34.457Z", - "EndTime": "2026-05-25T10:39:07.113Z", - "CreationTime": "2026-05-25T10:38:34.457Z" - }, - { - "Key": "f4f78847-7e40-4843-9179-fbf25b9b251b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T13:24:19.273Z", - "EndTime": "2026-05-22T13:24:20.907Z", - "CreationTime": "2026-05-22T13:24:19.273Z" - }, - { - "Key": "79e6d2c6-ab69-4e32-a831-d80f2f6036b6", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:26:21.197Z", - "EndTime": "2026-05-22T09:26:53.467Z", - "CreationTime": "2026-05-22T09:26:21.197Z" - }, - { - "Key": "4c063a0a-dd55-41c1-b2b3-00fbe83ee390", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T09:24:26.163Z", - "EndTime": "2026-05-22T09:24:59.140Z", - "CreationTime": "2026-05-22T09:24:26.163Z" - }, - { - "Key": "6eea1343-bceb-4784-8ad9-1164f6238c93", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-22T07:55:57.747Z", - "EndTime": "2026-05-22T07:56:11.837Z", - "CreationTime": "2026-05-22T07:55:57.747Z" - }, - { - "Key": "61f7af05-ad2a-464d-a23e-ba2b45fb59c1", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T13:05:49.550Z", - "EndTime": "2026-05-21T13:06:01.843Z", - "CreationTime": "2026-05-21T13:05:49.550Z" - }, - { - "Key": "49655b57-ad0a-410c-8256-38c215f9246d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-21T09:24:58.130Z", - "EndTime": "2026-05-21T09:25:02.813Z", - "CreationTime": "2026-05-21T09:24:58.130Z" - }, - { - "Key": "e92969a4-b6c1-49dc-9d25-1364ecef2e83", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:11:52.310Z", - "EndTime": "2026-05-20T08:13:01.203Z", - "CreationTime": "2026-05-20T08:11:52.310Z" - }, - { - "Key": "25407672-d7fa-4d30-8ea9-4b3e681e32de", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:08:26.657Z", - "EndTime": "2026-05-20T08:09:37.057Z", - "CreationTime": "2026-05-20T08:08:26.657Z" - }, - { - "Key": "72ba9624-b20f-4cf8-ba2d-c6c3f3781e77", - "State": "Successful", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-20T08:05:09.907Z", - "EndTime": "2026-05-20T08:06:18.867Z", - "CreationTime": "2026-05-20T08:05:09.907Z" - }, - { - "Key": "899a0c10-d81b-44ea-9d29-cecc00fd8976", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:21.587Z", - "EndTime": "2026-05-19T10:04:04.930Z", - "CreationTime": "2026-05-19T10:03:21.587Z" - }, - { - "Key": "8d632370-0b0c-4afb-ab4d-d50a5c2340d8", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "", - "MachineKey": "", - "StartTime": "2026-05-19T10:03:01.293Z", - "EndTime": "2026-05-19T10:03:01.293Z", - "CreationTime": "2026-05-19T10:02:52.723Z" - }, - { - "Key": "b682c801-bf93-488e-8c04-52bf0a31441d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:08:03.050Z", - "EndTime": "2026-05-19T08:08:11.767Z", - "CreationTime": "2026-05-19T08:08:03.050Z" - }, - { - "Key": "cedeff5a-ba9b-4ff5-80c2-b79b5662f59b", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T12:48:38.730Z", - "EndTime": "2026-05-18T12:49:13.423Z", - "CreationTime": "2026-05-18T12:48:38.730Z" - }, - { - "Key": "d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-15T17:09:44.470Z", - "EndTime": "2026-05-15T17:10:18.527Z", - "CreationTime": "2026-05-15T17:09:44.470Z" - }, - { - "Key": "7bb415a6-f17f-47e5-a43f-d1d38082eb8d", - "State": "Faulted", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-11T10:56:42.527Z", - "EndTime": "2026-05-11T10:56:43.227Z", - "CreationTime": "2026-05-11T10:56:42.527Z" - } - ], - "Pagination": { - "Returned": 23, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/525de3e550.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/525de3e550.json deleted file mode 100644 index 3b2dda53e6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/525de3e550.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/78433ce649.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/78433ce649.json deleted file mode 100644 index 3979ab6831..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/78433ce649.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "LicenseInfo", - "Data": { - "IsExpired": false, - "SubscriptionPlan": "ENTERPRISE", - "Allowed": { - "StudioPro": 0, - "BusinessRule": 0, - "Headless": 0, - "Development": 0, - "AppTest": 0, - "CaseManagement": 0, - "PerformanceTest": 0, - "ProcessOrchestration": 0, - "Attended": 0, - "Flow": 0, - "StudioX": 0, - "AgentService": 0, - "NonProduction": 0, - "TestAutomation": 1, - "Unattended": 1, - "Hosting": 0, - "Local": 0 - }, - "Used": { - "StudioPro": 0, - "BusinessRule": 0, - "Headless": 0, - "Development": 0, - "AppTest": 0, - "CaseManagement": 0, - "PerformanceTest": 0, - "ProcessOrchestration": 0, - "Attended": 0, - "Flow": 0, - "StudioX": 0, - "AgentService": 0, - "NonProduction": 0, - "TestAutomation": 0, - "Unattended": 1, - "Hosting": 0, - "Local": 0 - }, - "LicensedFeatures": [] - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/README.md deleted file mode 100644 index a2bdc2c047..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# No Login Pending — Faithful Replay - -This scenario replays a real UiPath troubleshooting investigation where the -agent reached a verified resolution. The fixtures are the verbatim `uip` -CLI responses captured from that session. - -## What the original session uncovered - -A user reported their last job in the Shared folder was stuck in `Pending` -and never started. The agent discovered exactly one Pending job there -(`0fd7bea5-…`, process ERN, Unattended) whose `PendingReasons.ErrorCodes` -was `["RobotNoMatchingUsernames"]` — the robot account's credential-store -username did not match any machine-user mapping on the eligible machine -template (`DanLaptopNew`). The fix is to align the robot credentials with -the machine user, assign a matching account to the folder, or run the job -under a different account. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -(No `process/` snapshot — this investigation is purely CLI-driven; no -project source was read.) - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- The agent's presented resolution names `RobotNoMatchingUsernames` / - credential-mismatch as the root cause and recommends one of the three - documented remediations in `robot-credentials.md`, without falling for - the no-host / stale-dispatch / license-exhaustion misreads - -## Re-running the extraction - -If the source transcript changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --transcript \ - --scenario-name no-login-pending --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/a1575f0339.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/a1575f0339.json deleted file mode 100644 index caaf9a9d04..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/a1575f0339.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "0fd7bea5-4182-43ed-8273-9c43410add3c", - "State": "Pending", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d4e90cf1-848e-474c-b20b-fe20d97b154a", - "CreationTime": "2026-05-27T16:24:24.603Z", - "ReleaseName": "ERN", - "Type": "Unattended", - "EnvironmentVariables": "", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "EntryPointPath": "EditorLink.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "", - "RemoteControlAccess": "None", - "AutopilotForRobots": { - "Enabled": false, - "HealingEnabled": false - }, - "Id": 65330689, - "PendingReasons": { - "IsValid": false, - "Errors": [], - "ErrorCodes": [ - "RobotNoMatchingUsernames" - ] - } - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/fda63e3155.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/fda63e3155.json deleted file mode 100644 index a32f48ae02..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/fda63e3155.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "Name": "DanLaptopNew", - "ID": 1890824, - "Key": "b8d82bd0-1d19-4a23-8b34-fc26927d4cfa", - "Type": "Template", - "Scope": "Default", - "Description": "", - "UnattendedSlots": 1, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2062" - }, - { - "Name": "[Default] Cloud Robots - Serverless", - "ID": 1104780, - "Key": "3b82fc7a-8154-4c17-ac05-60f026904b8b", - "Type": "Template", - "Scope": "Serverless", - "Description": "Automatically generated for use in Personal Workspaces", - "UnattendedSlots": 0, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "78d93cb4-23ae-4d30-9f56-2640d6584221" - }, - { - "Name": "original_email@test.com's workspace machine", - "ID": 961460, - "Key": "6f12f3e7-75f5-475e-aa34-3f5bd5c564e1", - "Type": "Template", - "Scope": "PersonalWorkspace", - "Description": "", - "UnattendedSlots": 1, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/manifest.json deleted file mode 100644 index 2d6714814d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/m/r/manifest.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls. Match strings intentionally omit `--output json` so the recorded discovery path resolves deterministically whether or not the agent appends the flag (substring match, first wins); without this, a missing-flag call falls through to the permissive empty-[] default and the agent abandons discovery.", - "rules": [ - { - "match": "or folders list", - "file": "525de3e550.json" - }, - { - "match": "or jobs list --folder-key defb8e05-e36b-4c36-bf11-0b4d08ce6cd1", - "file": "12363ea7c4.json" - }, - { - "match": "or jobs get 0fd7bea5-4182-43ed-8273-9c43410add3c", - "file": "a1575f0339.json" - }, - { - "match": "or machines list", - "file": "fda63e3155.json" - }, - { - "match": "or licenses info", - "file": "78433ce649.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/uip-fixture.json new file mode 100644 index 0000000000..5b8eda0104 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/data/uip-fixture.json @@ -0,0 +1,62 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "defb8e05-e36b-4c36-bf11-0b4d08ce6cd1" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Pending\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"\",\r\n \"EndTime\": \"\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\"\r\n },\r\n {\r\n \"Key\": \"6847a94d-68aa-4cfe-93dd-572a1c570185\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-27T16:19:15.317Z\",\r\n \"EndTime\": \"2026-05-27T16:20:00.493Z\",\r\n \"CreationTime\": \"2026-05-27T16:19:14.617Z\"\r\n },\r\n {\r\n \"Key\": \"3163478d-6d95-48cb-b447-a358af3813cc\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"BlankProcess1\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:50:22.467Z\",\r\n \"EndTime\": \"2026-05-26T08:50:57.280Z\",\r\n \"CreationTime\": \"2026-05-26T08:50:21.833Z\"\r\n },\r\n {\r\n \"Key\": \"777970ae-5c98-454f-85a2-3325dcc98f81\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"EndTime\": \"2026-05-26T08:48:20.617Z\",\r\n \"CreationTime\": \"2026-05-25T17:58:16.720Z\"\r\n },\r\n {\r\n \"Key\": \"66f6f28f-e41c-4660-aec4-374a4180ba28\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"EndTime\": \"2026-05-25T17:57:45.707Z\",\r\n \"CreationTime\": \"2026-05-25T17:57:32.123Z\"\r\n },\r\n {\r\n \"Key\": \"8c994220-a19a-4c62-9868-6df7b61c97e3\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T17:56:44.133Z\",\r\n \"EndTime\": \"2026-05-25T17:56:48.883Z\",\r\n \"CreationTime\": \"2026-05-25T17:56:43.687Z\"\r\n },\r\n {\r\n \"Key\": \"c5907f72-8a22-410b-9d66-260a3a4ec46d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T12:34:25.143Z\",\r\n \"EndTime\": \"2026-05-25T12:34:29.537Z\",\r\n \"CreationTime\": \"2026-05-25T12:34:25.143Z\"\r\n },\r\n {\r\n \"Key\": \"91d543d2-1b71-4233-a94a-a1389014c998\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-25T10:38:34.457Z\",\r\n \"EndTime\": \"2026-05-25T10:39:07.113Z\",\r\n \"CreationTime\": \"2026-05-25T10:38:34.457Z\"\r\n },\r\n {\r\n \"Key\": \"f4f78847-7e40-4843-9179-fbf25b9b251b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T13:24:19.273Z\",\r\n \"EndTime\": \"2026-05-22T13:24:20.907Z\",\r\n \"CreationTime\": \"2026-05-22T13:24:19.273Z\"\r\n },\r\n {\r\n \"Key\": \"79e6d2c6-ab69-4e32-a831-d80f2f6036b6\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:26:21.197Z\",\r\n \"EndTime\": \"2026-05-22T09:26:53.467Z\",\r\n \"CreationTime\": \"2026-05-22T09:26:21.197Z\"\r\n },\r\n {\r\n \"Key\": \"4c063a0a-dd55-41c1-b2b3-00fbe83ee390\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T09:24:26.163Z\",\r\n \"EndTime\": \"2026-05-22T09:24:59.140Z\",\r\n \"CreationTime\": \"2026-05-22T09:24:26.163Z\"\r\n },\r\n {\r\n \"Key\": \"6eea1343-bceb-4784-8ad9-1164f6238c93\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-22T07:55:57.747Z\",\r\n \"EndTime\": \"2026-05-22T07:56:11.837Z\",\r\n \"CreationTime\": \"2026-05-22T07:55:57.747Z\"\r\n },\r\n {\r\n \"Key\": \"61f7af05-ad2a-464d-a23e-ba2b45fb59c1\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T13:05:49.550Z\",\r\n \"EndTime\": \"2026-05-21T13:06:01.843Z\",\r\n \"CreationTime\": \"2026-05-21T13:05:49.550Z\"\r\n },\r\n {\r\n \"Key\": \"49655b57-ad0a-410c-8256-38c215f9246d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-21T09:24:58.130Z\",\r\n \"EndTime\": \"2026-05-21T09:25:02.813Z\",\r\n \"CreationTime\": \"2026-05-21T09:24:58.130Z\"\r\n },\r\n {\r\n \"Key\": \"e92969a4-b6c1-49dc-9d25-1364ecef2e83\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:11:52.310Z\",\r\n \"EndTime\": \"2026-05-20T08:13:01.203Z\",\r\n \"CreationTime\": \"2026-05-20T08:11:52.310Z\"\r\n },\r\n {\r\n \"Key\": \"25407672-d7fa-4d30-8ea9-4b3e681e32de\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:08:26.657Z\",\r\n \"EndTime\": \"2026-05-20T08:09:37.057Z\",\r\n \"CreationTime\": \"2026-05-20T08:08:26.657Z\"\r\n },\r\n {\r\n \"Key\": \"72ba9624-b20f-4cf8-ba2d-c6c3f3781e77\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-20T08:05:09.907Z\",\r\n \"EndTime\": \"2026-05-20T08:06:18.867Z\",\r\n \"CreationTime\": \"2026-05-20T08:05:09.907Z\"\r\n },\r\n {\r\n \"Key\": \"899a0c10-d81b-44ea-9d29-cecc00fd8976\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:21.587Z\",\r\n \"EndTime\": \"2026-05-19T10:04:04.930Z\",\r\n \"CreationTime\": \"2026-05-19T10:03:21.587Z\"\r\n },\r\n {\r\n \"Key\": \"8d632370-0b0c-4afb-ab4d-d50a5c2340d8\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"EndTime\": \"2026-05-19T10:03:01.293Z\",\r\n \"CreationTime\": \"2026-05-19T10:02:52.723Z\"\r\n },\r\n {\r\n \"Key\": \"b682c801-bf93-488e-8c04-52bf0a31441d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-19T08:08:03.050Z\",\r\n \"EndTime\": \"2026-05-19T08:08:11.767Z\",\r\n \"CreationTime\": \"2026-05-19T08:08:03.050Z\"\r\n },\r\n {\r\n \"Key\": \"cedeff5a-ba9b-4ff5-80c2-b79b5662f59b\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-18T12:48:38.730Z\",\r\n \"EndTime\": \"2026-05-18T12:49:13.423Z\",\r\n \"CreationTime\": \"2026-05-18T12:48:38.730Z\"\r\n },\r\n {\r\n \"Key\": \"d2c90d73-bcee-4f9d-b9fc-d37146b7f6ff\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-15T17:09:44.470Z\",\r\n \"EndTime\": \"2026-05-15T17:10:18.527Z\",\r\n \"CreationTime\": \"2026-05-15T17:09:44.470Z\"\r\n },\r\n {\r\n \"Key\": \"7bb415a6-f17f-47e5-a43f-d1d38082eb8d\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-11T10:56:42.527Z\",\r\n \"EndTime\": \"2026-05-11T10:56:43.227Z\",\r\n \"CreationTime\": \"2026-05-11T10:56:42.527Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 23,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "0fd7bea5-4182-43ed-8273-9c43410add3c" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"0fd7bea5-4182-43ed-8273-9c43410add3c\",\r\n \"State\": \"Pending\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d4e90cf1-848e-474c-b20b-fe20d97b154a\",\r\n \"CreationTime\": \"2026-05-27T16:24:24.603Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"EntryPointPath\": \"EditorLink.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": false,\r\n \"HealingEnabled\": false\r\n },\r\n \"Id\": 65330689,\r\n \"PendingReasons\": {\r\n \"IsValid\": false,\r\n \"Errors\": [],\r\n \"ErrorCodes\": [\r\n \"RobotNoMatchingUsernames\"\r\n ]\r\n }\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"3a1e81c2-07ca-4d6b-bc07-7e0c7b94ea3a\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"defb8e05-e36b-4c36-bf11-0b4d08ce6cd1\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"DanLaptopNew\",\r\n \"ID\": 1890824,\r\n \"Key\": \"b8d82bd0-1d19-4a23-8b34-fc26927d4cfa\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"Default\",\r\n \"Description\": \"\",\r\n \"UnattendedSlots\": 1,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2062\"\r\n },\r\n {\r\n \"Name\": \"[Default] Cloud Robots - Serverless\",\r\n \"ID\": 1104780,\r\n \"Key\": \"3b82fc7a-8154-4c17-ac05-60f026904b8b\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"Serverless\",\r\n \"Description\": \"Automatically generated for use in Personal Workspaces\",\r\n \"UnattendedSlots\": 0,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"78d93cb4-23ae-4d30-9f56-2640d6584221\"\r\n },\r\n {\r\n \"Name\": \"original_email@test.com's workspace machine\",\r\n \"ID\": 961460,\r\n \"Key\": \"6f12f3e7-75f5-475e-aa34-3f5bd5c564e1\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"PersonalWorkspace\",\r\n \"Description\": \"\",\r\n \"UnattendedSlots\": 1,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "licenses", + "info" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LicenseInfo\",\r\n \"Data\": {\r\n \"IsExpired\": false,\r\n \"SubscriptionPlan\": \"ENTERPRISE\",\r\n \"Allowed\": {\r\n \"StudioPro\": 0,\r\n \"BusinessRule\": 0,\r\n \"Headless\": 0,\r\n \"Development\": 0,\r\n \"AppTest\": 0,\r\n \"CaseManagement\": 0,\r\n \"PerformanceTest\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"Attended\": 0,\r\n \"Flow\": 0,\r\n \"StudioX\": 0,\r\n \"AgentService\": 0,\r\n \"NonProduction\": 0,\r\n \"TestAutomation\": 1,\r\n \"Unattended\": 1,\r\n \"Hosting\": 0,\r\n \"Local\": 0\r\n },\r\n \"Used\": {\r\n \"StudioPro\": 0,\r\n \"BusinessRule\": 0,\r\n \"Headless\": 0,\r\n \"Development\": 0,\r\n \"AppTest\": 0,\r\n \"CaseManagement\": 0,\r\n \"PerformanceTest\": 0,\r\n \"ProcessOrchestration\": 0,\r\n \"Attended\": 0,\r\n \"Flow\": 0,\r\n \"StudioX\": 0,\r\n \"AgentService\": 0,\r\n \"NonProduction\": 0,\r\n \"TestAutomation\": 0,\r\n \"Unattended\": 1,\r\n \"Hosting\": 0,\r\n \"Local\": 0\r\n },\r\n \"LicensedFeatures\": []\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/task.yaml index a7d159f9f3..32f65f5d79 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/no-login-pending/task.yaml @@ -21,20 +21,10 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/README.md new file mode 100644 index 0000000000..8795e2b146 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/README.md @@ -0,0 +1,58 @@ +# Platform Incident — Many Jobs Fault at Once + +Reproduces the `platform-incident-correlation` playbook: a burst of +failures across multiple unrelated processes and folders in a tight +window, with Orchestrator-side infrastructure errors — the +fingerprint of a UiPath platform incident, not a single automation's +bug. + +## What this scenario uncovers + +**Root Cause:** A platform incident around 08:12–08:17. Five distinct +processes across three folders (`NightlyBilling`/Billing, +`ClaimIntake`/ClaimsOps, `PayrollSync`/HRBots, +`InvoiceMatcher`/Billing, `OnboardingBot`/HRBots) all fault in a +5-minute window with `503 Service Unavailable` / `Could not obtain +the user token from Orchestrator` (`ErrorCode: Orchestrator`). No +shared automation dependency. The correct move is to correlate with +status.uipath.com / known-issues and monitor + rerun + report — not +debug the individual automations. + +Maps to: +`references/products/orchestrator/playbooks/platform-incident-correlation.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal project for one victim process (`NightlyBilling`) | + +> Fixtures authored from the playbook signature, not captured from a +> real session. Note the sandbox has no live status.uipath.com; the +> graded deliverable is the **recommendation** to correlate with the +> status page + known-issues and monitor/rerun/report. + +## Distinguishing fingerprint + +No single Info gives the answer — the diagnosis emerges only from the +**fan-out pattern** (many unrelated processes, three folders, one +short window, Orchestrator-side infra errors). An agent that latches +onto one job's 503 and debugs that automation scores low; one that +recognizes the platform-incident pattern and recommends status-page +correlation + monitor/rerun/report scores full. Inherently +bypass-resistant. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified a platform incident from the cross-folder fan-out + (not a single-automation bug) and recommended correlating with + status.uipath.com / known-issues, monitoring, rerunning after + recovery, and reporting via the customer portal — rather than + debugging the individual jobs. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/1aa6279d5b.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/1aa6279d5b.json deleted file mode 100644 index 828f269178..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/1aa6279d5b.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-26T08:12:04.880Z", "Message": "Job 'ddff7788-99aa-00bb-11cc-22dd33445566' failed to start. The remote server returned an error: (503) Server Unavailable."}, - {"Level": "Warn", "TimeStamp": "2026-06-26T08:12:04.100Z", "Message": "[Robot] Received HTTP 503 from Orchestrator gateway while requesting job start; upstream service temporarily unavailable."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/2b5d9064e1.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/2b5d9064e1.json deleted file mode 100644 index 24795d3ed3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/2b5d9064e1.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ddff7788-99aa-00bb-11cc-22dd33445577", - "StartTime": "2026-06-26T08:13:41.220Z", - "EndTime": "2026-06-26T08:13:42.010Z", - "State": "Faulted", - "Source": "Schedule", - "Info": "Could not obtain the user token from Orchestrator.", - "ReleaseName": "ClaimIntake", - "Type": "Unattended", - "RuntimeType": "Serverless", - "HostMachineName": "", - "FolderKey": "c6d7e8f9-a0b1-2345-6789-abcdef012345", - "ProcessType": "Process", - "JobError": {}, - "ErrorCode": "Orchestrator", - "TraceId": "ddff7788-99aa-00bb-11cc-22dd33445577", - "Id": 671770022 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/3c1c7d11e4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/3c1c7d11e4.json deleted file mode 100644 index 828f269178..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/3c1c7d11e4.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-26T08:12:04.880Z", "Message": "Job 'ddff7788-99aa-00bb-11cc-22dd33445566' failed to start. The remote server returned an error: (503) Server Unavailable."}, - {"Level": "Warn", "TimeStamp": "2026-06-26T08:12:04.100Z", "Message": "[Robot] Received HTTP 503 from Orchestrator gateway while requesting job start; upstream service temporarily unavailable."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/78f47f7072.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/78f47f7072.json deleted file mode 100644 index 4318c270de..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/78f47f7072.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ddff7788-99aa-00bb-11cc-22dd33445566", - "StartTime": "2026-06-26T08:12:03.100Z", - "EndTime": "2026-06-26T08:12:04.900Z", - "State": "Faulted", - "Source": "Schedule", - "Info": "The remote server returned an error: (503) Server Unavailable. Orchestrator returned 503 while starting the job.", - "ReleaseName": "NightlyBilling", - "Type": "Unattended", - "RuntimeType": "Unattended", - "HostMachineName": "MOCK-HOST", - "FolderKey": "b6c6d7e8-f9a0-1234-5678-9abcdef01234", - "ProcessType": "Process", - "JobError": {}, - "ErrorCode": "Orchestrator", - "TraceId": "ddff7788-99aa-00bb-11cc-22dd33445566", - "Id": 671770011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/README.md deleted file mode 100644 index 9b1f2884e9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/README.md +++ /dev/null @@ -1,57 +0,0 @@ -# Platform Incident — Many Jobs Fault at Once - -Reproduces the `platform-incident-correlation` playbook: a burst of -failures across multiple unrelated processes and folders in a tight -window, with Orchestrator-side infrastructure errors — the -fingerprint of a UiPath platform incident, not a single automation's -bug. - -## What this scenario uncovers - -**Root Cause:** A platform incident around 08:12–08:17. Five distinct -processes across three folders (`NightlyBilling`/Billing, -`ClaimIntake`/ClaimsOps, `PayrollSync`/HRBots, -`InvoiceMatcher`/Billing, `OnboardingBot`/HRBots) all fault in a -5-minute window with `503 Service Unavailable` / `Could not obtain -the user token from Orchestrator` (`ErrorCode: Orchestrator`). No -shared automation dependency. The correct move is to correlate with -status.uipath.com / known-issues and monitor + rerun + report — not -debug the individual automations. - -Maps to: -`references/products/orchestrator/playbooks/platform-incident-correlation.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal project for one victim process (`NightlyBilling`) | -| `data/m/r/*.json` | **synthetic** canned `uip` responses — a cross-folder `jobs list --state Faulted` burst, plus `jobs get` on two unrelated victims showing Orchestrator-side infra errors | -| `data/m/r/manifest.json` | dispatch table (folder NOT pinned — the fault is widespread) | - -> Fixtures authored from the playbook signature, not captured from a -> real session. Note the sandbox has no live status.uipath.com; the -> graded deliverable is the **recommendation** to correlate with the -> status page + known-issues and monitor/rerun/report. - -## Distinguishing fingerprint - -No single Info gives the answer — the diagnosis emerges only from the -**fan-out pattern** (many unrelated processes, three folders, one -short window, Orchestrator-side infra errors). An agent that latches -onto one job's 503 and debugs that automation scores low; one that -recognizes the platform-incident pattern and recommends status-page -correlation + monitor/rerun/report scores full. Inherently -bypass-resistant. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified a platform incident from the cross-folder fan-out - (not a single-automation bug) and recommended correlating with - status.uipath.com / known-issues, monitoring, rerunning after - recovery, and reporting via the customer portal — rather than - debugging the individual jobs. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/ee2628e6b6.json deleted file mode 100644 index 055af5a121..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - {"Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", "Name": "original_email@test.com's workspace", "Path": "original_email@test.com's workspace", "Type": "Personal", "ParentKey": ""}, - {"Key": "b6c6d7e8-f9a0-1234-5678-9abcdef01234", "Name": "Billing", "Path": "Billing", "Description": "Billing automations (NightlyBilling, InvoiceMatcher).", "Type": "Standard", "ParentKey": ""}, - {"Key": "c6d7e8f9-a0b1-2345-6789-abcdef012345", "Name": "ClaimsOps", "Path": "ClaimsOps", "Description": "Claims automations.", "Type": "Standard", "ParentKey": ""}, - {"Key": "d7e8f9a0-b1c2-3456-789a-bcdef0123456", "Name": "HRBots", "Path": "HRBots", "Description": "HR automations.", "Type": "Standard", "ParentKey": ""} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/f5f2f217c8.json deleted file mode 100644 index 51e03e2113..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - {"Key": "ddff7788-99aa-00bb-11cc-22dd33445566", "State": "Faulted", "ProcessName": "NightlyBilling", "FolderName": "Billing", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST", "StartTime": "2026-06-26T08:12:03.100Z", "EndTime": "2026-06-26T08:12:04.900Z"}, - {"Key": "ddff7788-99aa-00bb-11cc-22dd33445577", "State": "Faulted", "ProcessName": "ClaimIntake", "FolderName": "ClaimsOps", "Type": "Unattended", "RuntimeType": "Serverless", "Machine": "", "StartTime": "2026-06-26T08:13:41.220Z", "EndTime": "2026-06-26T08:13:42.010Z"}, - {"Key": "ddff7788-99aa-00bb-11cc-22dd33445588", "State": "Faulted", "ProcessName": "PayrollSync", "FolderName": "HRBots", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST-2", "StartTime": "2026-06-26T08:14:55.640Z", "EndTime": "2026-06-26T08:14:57.300Z"}, - {"Key": "ddff7788-99aa-00bb-11cc-22dd33445599", "State": "Faulted", "ProcessName": "InvoiceMatcher", "FolderName": "Billing", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST", "StartTime": "2026-06-26T08:15:30.880Z", "EndTime": "2026-06-26T08:15:32.140Z"}, - {"Key": "ddff7788-99aa-00bb-11cc-22dd334455aa", "State": "Faulted", "ProcessName": "OnboardingBot", "FolderName": "HRBots", "Type": "Unattended", "RuntimeType": "Serverless", "Machine": "", "StartTime": "2026-06-26T08:16:48.500Z", "EndTime": "2026-06-26T08:16:49.220Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/manifest.json deleted file mode 100644 index 50b561d6bc..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/m/r/manifest.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the platform-incident correlation scenario (platform-incident-correlation playbook). Folder is NOT pinned — the failure is widespread. The agent uses the cross-folder fallback: jobs list --state Faulted (no folder key) returns a burst of 5 unrelated processes across 3 folders in a 5-minute window with infra errors; jobs get on two samples (Billing/NightlyBilling 503, ClaimsOps/ClaimIntake token error) confirms different processes, same infra-error class. Conclusion: platform incident — correlate with status.uipath.com, monitor/rerun/report, do not debug individual jobs. Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --state Faulted", "file": "f5f2f217c8.json" }, - { "match": "or jobs get ddff7788-99aa-00bb-11cc-22dd33445566", "file": "78f47f7072.json" }, - { "match": "or jobs get ddff7788-99aa-00bb-11cc-22dd33445577", "file": "2b5d9064e1.json" }, - { "match": "or jobs logs ddff7788-99aa-00bb-11cc-22dd33445566 --level Error", "file": "3c1c7d11e4.json" }, - { "match": "or jobs logs ddff7788-99aa-00bb-11cc-22dd33445566", "file": "1aa6279d5b.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0 } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/uip-fixture.json new file mode 100644 index 0000000000..1b04d888b2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/data/uip-fixture.json @@ -0,0 +1,77 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "ddff7788-99aa-00bb-11cc-22dd33445566", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-26T08:12:04.880Z\", \"Message\": \"Job 'ddff7788-99aa-00bb-11cc-22dd33445566' failed to start. The remote server returned an error: (503) Server Unavailable.\"},\r\n {\"Level\": \"Warn\", \"TimeStamp\": \"2026-06-26T08:12:04.100Z\", \"Message\": \"[Robot] Received HTTP 503 from Orchestrator gateway while requesting job start; upstream service temporarily unavailable.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\"Key\": \"ddff7788-99aa-00bb-11cc-22dd33445566\", \"State\": \"Faulted\", \"ProcessName\": \"NightlyBilling\", \"FolderName\": \"Billing\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST\", \"StartTime\": \"2026-06-26T08:12:03.100Z\", \"EndTime\": \"2026-06-26T08:12:04.900Z\"},\r\n {\"Key\": \"ddff7788-99aa-00bb-11cc-22dd33445577\", \"State\": \"Faulted\", \"ProcessName\": \"ClaimIntake\", \"FolderName\": \"ClaimsOps\", \"Type\": \"Unattended\", \"RuntimeType\": \"Serverless\", \"Machine\": \"\", \"StartTime\": \"2026-06-26T08:13:41.220Z\", \"EndTime\": \"2026-06-26T08:13:42.010Z\"},\r\n {\"Key\": \"ddff7788-99aa-00bb-11cc-22dd33445588\", \"State\": \"Faulted\", \"ProcessName\": \"PayrollSync\", \"FolderName\": \"HRBots\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST-2\", \"StartTime\": \"2026-06-26T08:14:55.640Z\", \"EndTime\": \"2026-06-26T08:14:57.300Z\"},\r\n {\"Key\": \"ddff7788-99aa-00bb-11cc-22dd33445599\", \"State\": \"Faulted\", \"ProcessName\": \"InvoiceMatcher\", \"FolderName\": \"Billing\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST\", \"StartTime\": \"2026-06-26T08:15:30.880Z\", \"EndTime\": \"2026-06-26T08:15:32.140Z\"},\r\n {\"Key\": \"ddff7788-99aa-00bb-11cc-22dd334455aa\", \"State\": \"Faulted\", \"ProcessName\": \"OnboardingBot\", \"FolderName\": \"HRBots\", \"Type\": \"Unattended\", \"RuntimeType\": \"Serverless\", \"Machine\": \"\", \"StartTime\": \"2026-06-26T08:16:48.500Z\", \"EndTime\": \"2026-06-26T08:16:49.220Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ddff7788-99aa-00bb-11cc-22dd33445566" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ddff7788-99aa-00bb-11cc-22dd33445566\",\r\n \"StartTime\": \"2026-06-26T08:12:03.100Z\",\r\n \"EndTime\": \"2026-06-26T08:12:04.900Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Schedule\",\r\n \"Info\": \"The remote server returned an error: (503) Server Unavailable. Orchestrator returned 503 while starting the job.\",\r\n \"ReleaseName\": \"NightlyBilling\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"FolderKey\": \"b6c6d7e8-f9a0-1234-5678-9abcdef01234\",\r\n \"ProcessType\": \"Process\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Orchestrator\",\r\n \"TraceId\": \"ddff7788-99aa-00bb-11cc-22dd33445566\",\r\n \"Id\": 671770011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ddff7788-99aa-00bb-11cc-22dd33445577" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ddff7788-99aa-00bb-11cc-22dd33445577\",\r\n \"StartTime\": \"2026-06-26T08:13:41.220Z\",\r\n \"EndTime\": \"2026-06-26T08:13:42.010Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Schedule\",\r\n \"Info\": \"Could not obtain the user token from Orchestrator.\",\r\n \"ReleaseName\": \"ClaimIntake\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Serverless\",\r\n \"HostMachineName\": \"\",\r\n \"FolderKey\": \"c6d7e8f9-a0b1-2345-6789-abcdef012345\",\r\n \"ProcessType\": \"Process\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Orchestrator\",\r\n \"TraceId\": \"ddff7788-99aa-00bb-11cc-22dd33445577\",\r\n \"Id\": 671770022\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ddff7788-99aa-00bb-11cc-22dd33445566" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-26T08:12:04.880Z\", \"Message\": \"Job 'ddff7788-99aa-00bb-11cc-22dd33445566' failed to start. The remote server returned an error: (503) Server Unavailable.\"},\r\n {\"Level\": \"Warn\", \"TimeStamp\": \"2026-06-26T08:12:04.100Z\", \"Message\": \"[Robot] Received HTTP 503 from Orchestrator gateway while requesting job start; upstream service temporarily unavailable.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\", \"Name\": \"original_email@test.com's workspace\", \"Path\": \"original_email@test.com's workspace\", \"Type\": \"Personal\", \"ParentKey\": \"\"},\r\n {\"Key\": \"b6c6d7e8-f9a0-1234-5678-9abcdef01234\", \"Name\": \"Billing\", \"Path\": \"Billing\", \"Description\": \"Billing automations (NightlyBilling, InvoiceMatcher).\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"c6d7e8f9-a0b1-2345-6789-abcdef012345\", \"Name\": \"ClaimsOps\", \"Path\": \"ClaimsOps\", \"Description\": \"Claims automations.\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"d7e8f9a0-b1c2-3456-789a-bcdef0123456\", \"Name\": \"HRBots\", \"Path\": \"HRBots\", \"Description\": \"HR automations.\", \"Type\": \"Standard\", \"ParentKey\": \"\"}\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/task.yaml index 5e9db181f2..f61bf2bd3a 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/platform-incident-multi-process-fault/task.yaml @@ -28,21 +28,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/README.md new file mode 100644 index 0000000000..420068baf8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/README.md @@ -0,0 +1,36 @@ +# Queue Items Failing Test — Faithful Replay + +This scenario replays a real UiPath diagnostic investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +Root Cause: Invalid SpecificContent enqueued by an upstream dispatcher into the ItemsQueue queue (2 items with `Country='ZZ'`, 2 items with empty `Amount`). The performer correctly surfaced these as BusinessExceptions. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-diagnostics` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name queue-items-failing-test --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/04932177a0.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/04932177a0.json deleted file mode 100644 index 07533b8766..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/04932177a0.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "QueueList", - "Data": [ - { - "retentionAction": "Delete", - "retentionPeriod": 30, - "staleRetentionAction": "Delete", - "staleRetentionPeriod": 180, - "itemsToProcess": 0, - "itemsInProgress": 0, - "queueDefinitionId": 13713, - "queueDefinitionKey": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "queueDefinitionName": "ItemsQueue", - "queueDefinitionDescription": "", - "queueDefinitionAcceptAutomaticallyRetry": false, - "queueDefinitionRetryAbandonedItems": false, - "queueDefinitionMaxNumberOfRetries": 1, - "queueDefinitionEnforceUniqueReference": false, - "processingMeanTime": 0.265, - "successfulTransactionsNo": 4, - "applicationExceptionsNo": 0, - "businessExceptionsNo": 4, - "successfulTransactionsProcessingTime": 0.38, - "applicationExceptionsProcessingTime": 0, - "businessExceptionsProcessingTime": 0.15, - "totalNumberOfTransactions": 8, - "specificDataJsonSchemaExists": false, - "outputDataJsonSchemaExists": false, - "analyticsDataJsonSchemaExists": false, - "queueFoldersCount": 1, - "tags": [], - "id": 13713 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/05887ec63a.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/05887ec63a.json deleted file mode 100644 index ca2e24f562..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/05887ec63a.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Failure", - "Message": "Error listing queues", - "Instructions": "HTTP 400: Invalid OData query options." -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/5089f70e12.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/5089f70e12.json deleted file mode 100644 index ad45f77f28..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/5089f70e12.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "Result": "Success", - "Code": "QueueItem", - "Data": { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "tags": [], - "id": 13713 - }, - "processingException": { - "reason": "Unsupported country: ZZ", - "type": "BusinessException", - "creationTime": "2026-05-14T12:06:49.687Z" - }, - "error": { - "code": "User.QueueItemBusinessException", - "title": "Unsupported country: ZZ", - "detail": "", - "category": "User", - "timestamp": "2026-05-14T12:06:49.687Z" - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-006", - "Amount": "75.00", - "Country": "ZZ" - }, - "status": "Failed", - "reviewStatus": "None", - "key": "3b021051-08e0-4140-9448-ea073a4e7e66", - "processingExceptionType": "BusinessException", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:49.553Z", - "endProcessing": "2026-05-14T12:06:49.680Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "3b021051-08e0-4140-9448-ea073a4e7e66", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-006\",\"Amount\":\"75.00\",\"Country\":\"ZZ\"}}", - "creationTime": "2026-05-14T12:05:33.287Z", - "rowVersion": "AAAAAADkxLM=", - "organizationUnitId": 28927, - "id": 3569514 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/9a940defe7.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/9a940defe7.json deleted file mode 100644 index 95fef32174..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/9a940defe7.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "af31127b-1083-4304-930d-ad638022a248", - "StartTime": "2026-05-14T12:06:46.917Z", - "EndTime": "2026-05-14T12:06:50.357Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "af31127b-1083-4304-930d-ad638022a248", - "Info": "Job completed", - "CreationTime": "2026-05-14T12:06:46.917Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53412, - "EntryPointPath": "Transaction.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-14T12:06:50.357Z", - "TraceId": "af31127b10834304930dad638022a248", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 4014104 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/README.md deleted file mode 100644 index f49569b16c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Queue Items Failing Test — Faithful Replay - -This scenario replays a real UiPath diagnostic investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -Root Cause: Invalid SpecificContent enqueued by an upstream dispatcher into the ItemsQueue queue (2 items with `Country='ZZ'`, 2 items with empty `Amount`). The performer correctly surfaced these as BusinessExceptions. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-diagnostics` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name queue-items-failing-test --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/a382139095.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/a382139095.json deleted file mode 100644 index 1ead8e2a89..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/a382139095.json +++ /dev/null @@ -1,238 +0,0 @@ -{ - "Result": "Success", - "Code": "QueueItemList", - "Data": [ - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "processingException": { - "reason": "Unsupported country: ZZ", - "type": "BusinessException", - "creationTime": "2026-05-14T12:06:49.687Z" - }, - "error": { - "code": "User.QueueItemBusinessException", - "title": "Unsupported country: ZZ", - "detail": "", - "category": "User", - "timestamp": "2026-05-14T12:06:49.687Z" - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-006", - "Amount": "75.00", - "Country": "ZZ" - }, - "status": "Failed", - "reviewStatus": "None", - "key": "3b021051-08e0-4140-9448-ea073a4e7e66", - "processingExceptionType": "BusinessException", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:49.553Z", - "endProcessing": "2026-05-14T12:06:49.680Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "3b021051-08e0-4140-9448-ea073a4e7e66", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-006\",\"Amount\":\"75.00\",\"Country\":\"ZZ\"}}", - "creationTime": "2026-05-14T12:05:33.287Z", - "rowVersion": "AAAAAADkxLM=", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569514 - }, - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "processingException": { - "reason": "Unsupported country: ZZ", - "type": "BusinessException", - "creationTime": "2026-05-14T12:06:49.350Z" - }, - "error": { - "code": "User.QueueItemBusinessException", - "title": "Unsupported country: ZZ", - "detail": "", - "category": "User", - "timestamp": "2026-05-14T12:06:49.350Z" - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-005", - "Amount": "50.00", - "Country": "ZZ" - }, - "status": "Failed", - "reviewStatus": "None", - "key": "8164a35b-6b2b-4b91-bd0a-5909a9f3f0ea", - "processingExceptionType": "BusinessException", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:49.153Z", - "endProcessing": "2026-05-14T12:06:49.347Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "8164a35b-6b2b-4b91-bd0a-5909a9f3f0ea", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-005\",\"Amount\":\"50.00\",\"Country\":\"ZZ\"}}", - "creationTime": "2026-05-14T12:05:33.273Z", - "rowVersion": "AAAAAADkxLA=", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569513 - }, - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "processingException": { - "reason": "Transaction amount is null", - "type": "BusinessException", - "creationTime": "2026-05-14T12:06:48.970Z" - }, - "error": { - "code": "User.QueueItemBusinessException", - "title": "Transaction amount is null", - "detail": "", - "category": "User", - "timestamp": "2026-05-14T12:06:48.970Z" - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-004", - "Amount": "", - "Country": "UK" - }, - "status": "Failed", - "reviewStatus": "None", - "key": "746e754b-415a-4469-b588-368d6602e829", - "processingExceptionType": "BusinessException", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:48.837Z", - "endProcessing": "2026-05-14T12:06:48.963Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "746e754b-415a-4469-b588-368d6602e829", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-004\",\"Amount\":\"\",\"Country\":\"UK\"}}", - "creationTime": "2026-05-14T12:05:33.253Z", - "rowVersion": "AAAAAADkxKw=", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569512 - }, - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "processingException": { - "reason": "Transaction amount is null", - "type": "BusinessException", - "creationTime": "2026-05-14T12:06:48.577Z" - }, - "error": { - "code": "User.QueueItemBusinessException", - "title": "Transaction amount is null", - "detail": "", - "category": "User", - "timestamp": "2026-05-14T12:06:48.577Z" - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-003", - "Amount": "", - "Country": "US" - }, - "status": "Failed", - "reviewStatus": "None", - "key": "54333a46-7240-495e-8951-98f567855015", - "processingExceptionType": "BusinessException", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:48.403Z", - "endProcessing": "2026-05-14T12:06:48.570Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "54333a46-7240-495e-8951-98f567855015", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-003\",\"Amount\":\"\",\"Country\":\"US\"}}", - "creationTime": "2026-05-14T12:05:33.227Z", - "rowVersion": "AAAAAADkxKk=", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569511 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/c1b7f79f2c.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/c1b7f79f2c.json deleted file mode 100644 index 8bebe29473..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/c1b7f79f2c.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "3dffc6c6-c2f8-4432-af59-babc0e648595", - "State": "Stopped", - "ProcessName": "ERN", - "ReleaseKey": "", - "Source": "Agent", - "Type": "Attended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-14T12:06:21.083Z", - "EndTime": "2026-05-14T12:06:36.000Z", - "CreationTime": "2026-05-14T12:06:21.083Z" - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/d50678bc6c.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/d50678bc6c.json deleted file mode 100644 index c59a18343c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/d50678bc6c.json +++ /dev/null @@ -1,186 +0,0 @@ -{ - "Result": "Success", - "Code": "QueueItemList", - "Data": [ - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-002", - "Amount": "80.00", - "Country": "UK" - }, - "status": "Successful", - "reviewStatus": "None", - "key": "edb3a0e2-c428-4870-b368-febb6e94cd3e", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:48.110Z", - "endProcessing": "2026-05-14T12:06:48.243Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "edb3a0e2-c428-4870-b368-febb6e94cd3e", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-002\",\"Amount\":\"80.00\",\"Country\":\"UK\"}}", - "creationTime": "2026-05-14T12:05:33.213Z", - "rowVersion": "AAAAAADkxKY=", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569510 - }, - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-001", - "Amount": "120.50", - "Country": "US" - }, - "status": "Successful", - "reviewStatus": "None", - "key": "6a25007d-89aa-45a3-a9e1-34aba630c0ac", - "priority": "High", - "robotId": 6979, - "executorJobKey": "af31127b-1083-4304-930d-ad638022a248", - "startProcessing": "2026-05-14T12:06:47.577Z", - "endProcessing": "2026-05-14T12:06:47.930Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "6a25007d-89aa-45a3-a9e1-34aba630c0ac", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-001\",\"Amount\":\"120.50\",\"Country\":\"US\"}}", - "creationTime": "2026-05-14T12:05:33.200Z", - "rowVersion": "AAAAAADkxKQ=", - "operationId": "a0659d8cbfa94872a30f165dedbb813c-8de1e94fc60b4cdbb5d70e77b09388ba", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569509 - }, - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-002", - "Amount": "80.00", - "Country": "UK" - }, - "status": "Successful", - "reviewStatus": "None", - "key": "078b6002-27a9-404e-a101-27f6b7cbfddf", - "priority": "High", - "robotId": 6979, - "startProcessing": "2026-05-14T12:04:24.380Z", - "endProcessing": "2026-05-14T12:04:24.673Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "078b6002-27a9-404e-a101-27f6b7cbfddf", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-002\",\"Amount\":\"80.00\",\"Country\":\"UK\"}}", - "creationTime": "2026-05-14T12:04:15.423Z", - "rowVersion": "AAAAAADkxJQ=", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569504 - }, - { - "queueDefinitionId": 13713, - "queueDefinition": { - "key": "36e2cd45-f47f-4850-9a94-ac5f723faf3c", - "name": "ItemsQueue", - "description": "", - "maxNumberOfRetries": 1, - "acceptAutomaticallyRetry": true, - "retryAbandonedItems": false, - "enforceUniqueReference": false, - "encrypted": false, - "creationTime": "2026-05-14T11:25:32.653Z", - "slaInMinutes": 0, - "riskSlaInMinutes": 0, - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "tags": [], - "id": 13713 - }, - "encrypted": false, - "specificContent": { - "InvoiceId": "INV-001", - "Amount": "120.50", - "Country": "US" - }, - "status": "Successful", - "reviewStatus": "None", - "key": "4daa4853-3385-4460-ad65-ec9848391dea", - "priority": "High", - "robotId": 6979, - "startProcessing": "2026-05-14T12:04:23.437Z", - "endProcessing": "2026-05-14T12:04:24.177Z", - "secondsInPreviousAttempts": 0, - "retryNumber": 0, - "manualRetryNumber": 0, - "uniqueKey": "4daa4853-3385-4460-ad65-ec9848391dea", - "hasVideoRecorded": false, - "specificData": "{\"DynamicProperties\":{\"InvoiceId\":\"INV-001\",\"Amount\":\"120.50\",\"Country\":\"US\"}}", - "creationTime": "2026-05-14T12:04:15.407Z", - "rowVersion": "AAAAAADkxJI=", - "operationId": "da87a14d0a8e4c77aaf8404d62c8701c-5639bf3cbb6a43d0a4a0dfc2553c58f2", - "organizationUnitId": 28927, - "organizationUnitFullyQualifiedName": "Shared", - "id": 3569503 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/e3732af0be.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/e3732af0be.json deleted file mode 100644 index 873968e351..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/e3732af0be.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "QueueList", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/manifest.json deleted file mode 100644 index 1ec4fe686d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/m/r/manifest.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py \u2014 review fixtures + add expected_calls.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough — query strings vary between runs and canned responses go stale. Cached per-query within a run." - }, - { - "match": "or folders list --output json", - "file": "897a8be769.json" - }, - { - "match": "or queues list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --name ItemsQueue --output json", - "file": "05887ec63a.json" - }, - { - "match": "or queues list --folder-key dea5d8d6-ca39-4dbf-b4a5-3516bad38866 --name ItemsQueue --output json", - "file": "05887ec63a.json" - }, - { - "match": "or queues list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --output json", - "file": "04932177a0.json" - }, - { - "match": "or queues list --folder-key dea5d8d6-ca39-4dbf-b4a5-3516bad38866 --output json", - "file": "e3732af0be.json" - }, - { - "match": "or queue-items list --queue-definition-key 36e2cd45-f47f-4850-9a94-ac5f723faf3c --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --status Failed --output json", - "file": "a382139095.json" - }, - { - "match": "or queue-items list --queue-name ItemsQueue --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --status Failed --output json", - "file": "a382139095.json" - }, - { - "match": "or queue-items list --queue-name ItemsQueue --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --status Successful --limit 3 --output json", - "file": "d50678bc6c.json" - }, - { - "match": "or queue-items list --queue-name ItemsQueue --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --status Successful --output json", - "file": "d50678bc6c.json" - }, - { - "match": "or queue-items get 3b021051-08e0-4140-9448-ea073a4e7e66 --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --output json", - "file": "5089f70e12.json" - }, - { - "match": "or jobs get af31127b-1083-4304-930d-ad638022a248 --output json", - "file": "9a940defe7.json" - }, - { - "match": "or jobs list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --created-after 2026-05-14T12:00:00Z --created-before 2026-05-14T12:06:30Z --output json", - "file": "c1b7f79f2c.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/uip-fixture.json new file mode 100644 index 0000000000..5c1a83cc16 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/data/uip-fixture.json @@ -0,0 +1,201 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "queue-items", + "list", + "--queue-name", + "ItemsQueue", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--status", + "Successful", + "--limit", + "3", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueItemList\",\r\n \"Data\": [\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-002\",\r\n \"Amount\": \"80.00\",\r\n \"Country\": \"UK\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"edb3a0e2-c428-4870-b368-febb6e94cd3e\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:48.110Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:48.243Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"edb3a0e2-c428-4870-b368-febb6e94cd3e\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-002\\\",\\\"Amount\\\":\\\"80.00\\\",\\\"Country\\\":\\\"UK\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.213Z\",\r\n \"rowVersion\": \"AAAAAADkxKY=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569510\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-001\",\r\n \"Amount\": \"120.50\",\r\n \"Country\": \"US\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"6a25007d-89aa-45a3-a9e1-34aba630c0ac\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:47.577Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:47.930Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"6a25007d-89aa-45a3-a9e1-34aba630c0ac\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-001\\\",\\\"Amount\\\":\\\"120.50\\\",\\\"Country\\\":\\\"US\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.200Z\",\r\n \"rowVersion\": \"AAAAAADkxKQ=\",\r\n \"operationId\": \"a0659d8cbfa94872a30f165dedbb813c-8de1e94fc60b4cdbb5d70e77b09388ba\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569509\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-002\",\r\n \"Amount\": \"80.00\",\r\n \"Country\": \"UK\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"078b6002-27a9-404e-a101-27f6b7cbfddf\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"startProcessing\": \"2026-05-14T12:04:24.380Z\",\r\n \"endProcessing\": \"2026-05-14T12:04:24.673Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"078b6002-27a9-404e-a101-27f6b7cbfddf\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-002\\\",\\\"Amount\\\":\\\"80.00\\\",\\\"Country\\\":\\\"UK\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:04:15.423Z\",\r\n \"rowVersion\": \"AAAAAADkxJQ=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569504\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-001\",\r\n \"Amount\": \"120.50\",\r\n \"Country\": \"US\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"4daa4853-3385-4460-ad65-ec9848391dea\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"startProcessing\": \"2026-05-14T12:04:23.437Z\",\r\n \"endProcessing\": \"2026-05-14T12:04:24.177Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"4daa4853-3385-4460-ad65-ec9848391dea\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-001\\\",\\\"Amount\\\":\\\"120.50\\\",\\\"Country\\\":\\\"US\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:04:15.407Z\",\r\n \"rowVersion\": \"AAAAAADkxJI=\",\r\n \"operationId\": \"da87a14d0a8e4c77aaf8404d62c8701c-5639bf3cbb6a43d0a4a0dfc2553c58f2\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569503\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "queue-items", + "list", + "--queue-definition-key", + "36e2cd45-f47f-4850-9a94-ac5f723faf3c", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--status", + "Failed", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueItemList\",\r\n \"Data\": [\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Unsupported country: ZZ\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:49.687Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Unsupported country: ZZ\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:49.687Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-006\",\r\n \"Amount\": \"75.00\",\r\n \"Country\": \"ZZ\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"3b021051-08e0-4140-9448-ea073a4e7e66\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:49.553Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:49.680Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"3b021051-08e0-4140-9448-ea073a4e7e66\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-006\\\",\\\"Amount\\\":\\\"75.00\\\",\\\"Country\\\":\\\"ZZ\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.287Z\",\r\n \"rowVersion\": \"AAAAAADkxLM=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569514\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Unsupported country: ZZ\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:49.350Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Unsupported country: ZZ\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:49.350Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-005\",\r\n \"Amount\": \"50.00\",\r\n \"Country\": \"ZZ\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"8164a35b-6b2b-4b91-bd0a-5909a9f3f0ea\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:49.153Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:49.347Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"8164a35b-6b2b-4b91-bd0a-5909a9f3f0ea\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-005\\\",\\\"Amount\\\":\\\"50.00\\\",\\\"Country\\\":\\\"ZZ\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.273Z\",\r\n \"rowVersion\": \"AAAAAADkxLA=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569513\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Transaction amount is null\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:48.970Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Transaction amount is null\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:48.970Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-004\",\r\n \"Amount\": \"\",\r\n \"Country\": \"UK\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"746e754b-415a-4469-b588-368d6602e829\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:48.837Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:48.963Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"746e754b-415a-4469-b588-368d6602e829\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-004\\\",\\\"Amount\\\":\\\"\\\",\\\"Country\\\":\\\"UK\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.253Z\",\r\n \"rowVersion\": \"AAAAAADkxKw=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569512\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Transaction amount is null\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:48.577Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Transaction amount is null\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:48.577Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-003\",\r\n \"Amount\": \"\",\r\n \"Country\": \"US\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"54333a46-7240-495e-8951-98f567855015\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:48.403Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:48.570Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"54333a46-7240-495e-8951-98f567855015\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-003\\\",\\\"Amount\\\":\\\"\\\",\\\"Country\\\":\\\"US\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.227Z\",\r\n \"rowVersion\": \"AAAAAADkxKk=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569511\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "queue-items", + "list", + "--queue-name", + "ItemsQueue", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--status", + "Failed", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueItemList\",\r\n \"Data\": [\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Unsupported country: ZZ\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:49.687Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Unsupported country: ZZ\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:49.687Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-006\",\r\n \"Amount\": \"75.00\",\r\n \"Country\": \"ZZ\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"3b021051-08e0-4140-9448-ea073a4e7e66\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:49.553Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:49.680Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"3b021051-08e0-4140-9448-ea073a4e7e66\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-006\\\",\\\"Amount\\\":\\\"75.00\\\",\\\"Country\\\":\\\"ZZ\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.287Z\",\r\n \"rowVersion\": \"AAAAAADkxLM=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569514\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Unsupported country: ZZ\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:49.350Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Unsupported country: ZZ\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:49.350Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-005\",\r\n \"Amount\": \"50.00\",\r\n \"Country\": \"ZZ\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"8164a35b-6b2b-4b91-bd0a-5909a9f3f0ea\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:49.153Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:49.347Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"8164a35b-6b2b-4b91-bd0a-5909a9f3f0ea\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-005\\\",\\\"Amount\\\":\\\"50.00\\\",\\\"Country\\\":\\\"ZZ\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.273Z\",\r\n \"rowVersion\": \"AAAAAADkxLA=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569513\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Transaction amount is null\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:48.970Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Transaction amount is null\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:48.970Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-004\",\r\n \"Amount\": \"\",\r\n \"Country\": \"UK\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"746e754b-415a-4469-b588-368d6602e829\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:48.837Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:48.963Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"746e754b-415a-4469-b588-368d6602e829\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-004\\\",\\\"Amount\\\":\\\"\\\",\\\"Country\\\":\\\"UK\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.253Z\",\r\n \"rowVersion\": \"AAAAAADkxKw=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569512\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Transaction amount is null\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:48.577Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Transaction amount is null\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:48.577Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-003\",\r\n \"Amount\": \"\",\r\n \"Country\": \"US\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"54333a46-7240-495e-8951-98f567855015\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:48.403Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:48.570Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"54333a46-7240-495e-8951-98f567855015\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-003\\\",\\\"Amount\\\":\\\"\\\",\\\"Country\\\":\\\"US\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.227Z\",\r\n \"rowVersion\": \"AAAAAADkxKk=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569511\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "queue-items", + "list", + "--queue-name", + "ItemsQueue", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--status", + "Successful", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueItemList\",\r\n \"Data\": [\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-002\",\r\n \"Amount\": \"80.00\",\r\n \"Country\": \"UK\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"edb3a0e2-c428-4870-b368-febb6e94cd3e\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:48.110Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:48.243Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"edb3a0e2-c428-4870-b368-febb6e94cd3e\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-002\\\",\\\"Amount\\\":\\\"80.00\\\",\\\"Country\\\":\\\"UK\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.213Z\",\r\n \"rowVersion\": \"AAAAAADkxKY=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569510\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-001\",\r\n \"Amount\": \"120.50\",\r\n \"Country\": \"US\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"6a25007d-89aa-45a3-a9e1-34aba630c0ac\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:47.577Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:47.930Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"6a25007d-89aa-45a3-a9e1-34aba630c0ac\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-001\\\",\\\"Amount\\\":\\\"120.50\\\",\\\"Country\\\":\\\"US\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.200Z\",\r\n \"rowVersion\": \"AAAAAADkxKQ=\",\r\n \"operationId\": \"a0659d8cbfa94872a30f165dedbb813c-8de1e94fc60b4cdbb5d70e77b09388ba\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569509\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-002\",\r\n \"Amount\": \"80.00\",\r\n \"Country\": \"UK\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"078b6002-27a9-404e-a101-27f6b7cbfddf\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"startProcessing\": \"2026-05-14T12:04:24.380Z\",\r\n \"endProcessing\": \"2026-05-14T12:04:24.673Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"078b6002-27a9-404e-a101-27f6b7cbfddf\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-002\\\",\\\"Amount\\\":\\\"80.00\\\",\\\"Country\\\":\\\"UK\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:04:15.423Z\",\r\n \"rowVersion\": \"AAAAAADkxJQ=\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569504\r\n },\r\n {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-001\",\r\n \"Amount\": \"120.50\",\r\n \"Country\": \"US\"\r\n },\r\n \"status\": \"Successful\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"4daa4853-3385-4460-ad65-ec9848391dea\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"startProcessing\": \"2026-05-14T12:04:23.437Z\",\r\n \"endProcessing\": \"2026-05-14T12:04:24.177Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"4daa4853-3385-4460-ad65-ec9848391dea\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-001\\\",\\\"Amount\\\":\\\"120.50\\\",\\\"Country\\\":\\\"US\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:04:15.407Z\",\r\n \"rowVersion\": \"AAAAAADkxJI=\",\r\n \"operationId\": \"da87a14d0a8e4c77aaf8404d62c8701c-5639bf3cbb6a43d0a4a0dfc2553c58f2\",\r\n \"organizationUnitId\": 28927,\r\n \"organizationUnitFullyQualifiedName\": \"Shared\",\r\n \"id\": 3569503\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--created-after", + "2026-05-14T12:00:00Z", + "--created-before", + "2026-05-14T12:06:30Z", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"3dffc6c6-c2f8-4432-af59-babc0e648595\",\r\n \"State\": \"Stopped\",\r\n \"ProcessName\": \"ERN\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Agent\",\r\n \"Type\": \"Attended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-14T12:06:21.083Z\",\r\n \"EndTime\": \"2026-05-14T12:06:36.000Z\",\r\n \"CreationTime\": \"2026-05-14T12:06:21.083Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "queues", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--name", + "ItemsQueue", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error listing queues\",\r\n \"Instructions\": \"HTTP 400: Invalid OData query options.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "queues", + "list", + "--folder-key", + "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", + "--name", + "ItemsQueue", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Failure\",\r\n \"Message\": \"Error listing queues\",\r\n \"Instructions\": \"HTTP 400: Invalid OData query options.\"\r\n}\r\n" + }, + { + "argv": [ + "or", + "queue-items", + "get", + "3b021051-08e0-4140-9448-ea073a4e7e66", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueItem\",\r\n \"Data\": {\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinition\": {\r\n \"key\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"name\": \"ItemsQueue\",\r\n \"description\": \"\",\r\n \"maxNumberOfRetries\": 1,\r\n \"acceptAutomaticallyRetry\": true,\r\n \"retryAbandonedItems\": false,\r\n \"enforceUniqueReference\": false,\r\n \"encrypted\": false,\r\n \"creationTime\": \"2026-05-14T11:25:32.653Z\",\r\n \"slaInMinutes\": 0,\r\n \"riskSlaInMinutes\": 0,\r\n \"organizationUnitId\": 28927,\r\n \"tags\": [],\r\n \"id\": 13713\r\n },\r\n \"processingException\": {\r\n \"reason\": \"Unsupported country: ZZ\",\r\n \"type\": \"BusinessException\",\r\n \"creationTime\": \"2026-05-14T12:06:49.687Z\"\r\n },\r\n \"error\": {\r\n \"code\": \"User.QueueItemBusinessException\",\r\n \"title\": \"Unsupported country: ZZ\",\r\n \"detail\": \"\",\r\n \"category\": \"User\",\r\n \"timestamp\": \"2026-05-14T12:06:49.687Z\"\r\n },\r\n \"encrypted\": false,\r\n \"specificContent\": {\r\n \"InvoiceId\": \"INV-006\",\r\n \"Amount\": \"75.00\",\r\n \"Country\": \"ZZ\"\r\n },\r\n \"status\": \"Failed\",\r\n \"reviewStatus\": \"None\",\r\n \"key\": \"3b021051-08e0-4140-9448-ea073a4e7e66\",\r\n \"processingExceptionType\": \"BusinessException\",\r\n \"priority\": \"High\",\r\n \"robotId\": 6979,\r\n \"executorJobKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"startProcessing\": \"2026-05-14T12:06:49.553Z\",\r\n \"endProcessing\": \"2026-05-14T12:06:49.680Z\",\r\n \"secondsInPreviousAttempts\": 0,\r\n \"retryNumber\": 0,\r\n \"manualRetryNumber\": 0,\r\n \"uniqueKey\": \"3b021051-08e0-4140-9448-ea073a4e7e66\",\r\n \"hasVideoRecorded\": false,\r\n \"specificData\": \"{\\\"DynamicProperties\\\":{\\\"InvoiceId\\\":\\\"INV-006\\\",\\\"Amount\\\":\\\"75.00\\\",\\\"Country\\\":\\\"ZZ\\\"}}\",\r\n \"creationTime\": \"2026-05-14T12:05:33.287Z\",\r\n \"rowVersion\": \"AAAAAADkxLM=\",\r\n \"organizationUnitId\": 28927,\r\n \"id\": 3569514\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "queues", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueList\",\r\n \"Data\": [\r\n {\r\n \"retentionAction\": \"Delete\",\r\n \"retentionPeriod\": 30,\r\n \"staleRetentionAction\": \"Delete\",\r\n \"staleRetentionPeriod\": 180,\r\n \"itemsToProcess\": 0,\r\n \"itemsInProgress\": 0,\r\n \"queueDefinitionId\": 13713,\r\n \"queueDefinitionKey\": \"36e2cd45-f47f-4850-9a94-ac5f723faf3c\",\r\n \"queueDefinitionName\": \"ItemsQueue\",\r\n \"queueDefinitionDescription\": \"\",\r\n \"queueDefinitionAcceptAutomaticallyRetry\": false,\r\n \"queueDefinitionRetryAbandonedItems\": false,\r\n \"queueDefinitionMaxNumberOfRetries\": 1,\r\n \"queueDefinitionEnforceUniqueReference\": false,\r\n \"processingMeanTime\": 0.265,\r\n \"successfulTransactionsNo\": 4,\r\n \"applicationExceptionsNo\": 0,\r\n \"businessExceptionsNo\": 4,\r\n \"successfulTransactionsProcessingTime\": 0.38,\r\n \"applicationExceptionsProcessingTime\": 0,\r\n \"businessExceptionsProcessingTime\": 0.15,\r\n \"totalNumberOfTransactions\": 8,\r\n \"specificDataJsonSchemaExists\": false,\r\n \"outputDataJsonSchemaExists\": false,\r\n \"analyticsDataJsonSchemaExists\": false,\r\n \"queueFoldersCount\": 1,\r\n \"tags\": [],\r\n \"id\": 13713\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "queues", + "list", + "--folder-key", + "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"QueueList\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "af31127b-1083-4304-930d-ad638022a248", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"StartTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"EndTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"af31127b-1083-4304-930d-ad638022a248\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-05-14T12:06:46.917Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53412,\r\n \"EntryPointPath\": \"Transaction.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-14T12:06:50.357Z\",\r\n \"TraceId\": \"af31127b10834304930dad638022a248\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 4014104\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/task.yaml index 8af83ca0f3..240dafcf64 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/queue-items-failing-test/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - # Prepend ./mocks to the agent's PATH so bare `uip` resolves to the mock. - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/README.md new file mode 100644 index 0000000000..3d1e49e499 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/README.md @@ -0,0 +1,55 @@ +# Maximum Consecutive System Exceptions Reached + +Reproduces the `job-consecutive-system-exceptions` playbook: a +REFramework job aborts on its consecutive-System-Exception ceiling, +but the abort is a **symptom** — the real cause is the exception that +recurs on every transaction. + +``` +The maximum number of consecutive system exceptions was reached. +Consecutive retry number: 5. +``` + +## What this scenario uncovers + +**Root Cause:** A `SelectorNotFoundException` on the "Click Submit" +activity is thrown on every transaction (the order-portal UI +changed). After 5 consecutive system exceptions the framework +aborts. The graded diagnosis is the **recurring selector failure**, +not the threshold — and the fix is to repair the selector, not to +raise `MaxConsecutiveSystemExceptions`. + +Maps to: +`references/products/orchestrator/playbooks/job-consecutive-system-exceptions.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal REFramework-style UiPath project | + +> Fixtures authored from the playbook signature, not captured from a +> real session. + +## Distinguishing fingerprint + +The job Info gives only the abort message — the root cause is +reachable only by reading the logs/traces and spotting the identical +`SelectorNotFoundException` repeated per transaction. This makes the +scenario resist a "read the Info and answer" bypass: an agent that +stops at the abort message (or suggests raising the threshold) scores +low; one that finds the recurring selector failure scores full. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the recurring "Click Submit" SelectorNotFound- + Exception as the root cause (not the threshold) and recommended + fixing the selector (re-indicate / Object Repository / anchor + + Check State) rather than raising MaxConsecutiveSystemExceptions. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/0ea71af0c9.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/0ea71af0c9.json deleted file mode 100644 index 918401f78b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/0ea71af0c9.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-25T07:02:14.870Z", "Message": "System exception. The maximum number of consecutive system exceptions was reached. Consecutive retry number: 5. Aborting the process."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:02:10.300Z", "Message": "Transaction 5 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:01:44.210Z", "Message": "Transaction 4 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:01:18.050Z", "Message": "Transaction 3 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:00:52.900Z", "Message": "Transaction 2 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:00:27.640Z", "Message": "Transaction 1 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/12d7599762.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/12d7599762.json deleted file mode 100644 index 1405536b99..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/12d7599762.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "ccee6677-88ff-99aa-00bb-11cc22334455", - "State": "Faulted", - "ProcessName": "OrderReconciliation", - "Source": "Schedule", - "Type": "Unattended", - "RuntimeType": "Unattended", - "Machine": "MOCK-HOST", - "StartTime": "2026-06-25T07:00:01.200Z", - "EndTime": "2026-06-25T07:02:14.880Z", - "CreationTime": "2026-06-25T07:00:01.000Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/6d148edb25.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/6d148edb25.json deleted file mode 100644 index fb2f6ef814..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/6d148edb25.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - {"Activity": "Process Transaction (txn 1)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 2)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 3)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 4)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 5)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/89dd6ac5f6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/89dd6ac5f6.json deleted file mode 100644 index fb2f6ef814..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/89dd6ac5f6.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - {"Activity": "Process Transaction (txn 1)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 2)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 3)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 4)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}}, - {"Activity": "Process Transaction (txn 5)", "Status": "Faulted", "Error": {"Type": "UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException", "Activity": "Click Submit", "Message": "Could not find the UI element for 'Click Submit'."}} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/README.md deleted file mode 100644 index f2b0e42a1d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Maximum Consecutive System Exceptions Reached - -Reproduces the `job-consecutive-system-exceptions` playbook: a -REFramework job aborts on its consecutive-System-Exception ceiling, -but the abort is a **symptom** — the real cause is the exception that -recurs on every transaction. - -``` -The maximum number of consecutive system exceptions was reached. -Consecutive retry number: 5. -``` - -## What this scenario uncovers - -**Root Cause:** A `SelectorNotFoundException` on the "Click Submit" -activity is thrown on every transaction (the order-portal UI -changed). After 5 consecutive system exceptions the framework -aborts. The graded diagnosis is the **recurring selector failure**, -not the threshold — and the fix is to repair the selector, not to -raise `MaxConsecutiveSystemExceptions`. - -Maps to: -`references/products/orchestrator/playbooks/job-consecutive-system-exceptions.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal REFramework-style UiPath project | -| `data/m/r/*.json` | **synthetic** canned `uip` responses — logs carry the recurring SelectorNotFoundException (5×) before the abort; traces show "Click Submit" faulting every transaction | -| `data/m/r/manifest.json` | dispatch table | - -> Fixtures authored from the playbook signature, not captured from a -> real session. - -## Distinguishing fingerprint - -The job Info gives only the abort message — the root cause is -reachable only by reading the logs/traces and spotting the identical -`SelectorNotFoundException` repeated per transaction. This makes the -scenario resist a "read the Info and answer" bypass: an agent that -stops at the abort message (or suggests raising the threshold) scores -low; one that finds the recurring selector failure scores full. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the recurring "Click Submit" SelectorNotFound- - Exception as the root cause (not the threshold) and recommended - fixing the selector (re-indicate / Object Repository / anchor + - Check State) rather than raising MaxConsecutiveSystemExceptions. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/a812bb46c2.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/a812bb46c2.json deleted file mode 100644 index 918401f78b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/a812bb46c2.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-25T07:02:14.870Z", "Message": "System exception. The maximum number of consecutive system exceptions was reached. Consecutive retry number: 5. Aborting the process."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:02:10.300Z", "Message": "Transaction 5 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:01:44.210Z", "Message": "Transaction 4 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:01:18.050Z", "Message": "Transaction 3 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:00:52.900Z", "Message": "Transaction 2 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."}, - {"Level": "Error", "TimeStamp": "2026-06-25T07:00:27.640Z", "Message": "Transaction 1 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/aa692ef350.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/aa692ef350.json deleted file mode 100644 index 1e73494a21..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/aa692ef350.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ccee6677-88ff-99aa-00bb-11cc22334455", - "StartTime": "2026-06-25T07:00:01.200Z", - "EndTime": "2026-06-25T07:02:14.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "The maximum number of consecutive system exceptions was reached. Consecutive retry number: 5.", - "CreationTime": "2026-06-25T07:00:01.000Z", - "ReleaseName": "OrderReconciliation", - "Type": "Unattended", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "OrganizationUnitId": 7791635, - "FolderKey": "a5b5c6d7-e8f9-0123-4567-89abcdef0123", - "ProcessType": "Process", - "LastModificationTime": "2026-06-25T07:02:14.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "ccee6677-88ff-99aa-00bb-11cc22334455", - "Id": 671660011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/ee2628e6b6.json deleted file mode 100644 index f24d8fa21a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - {"Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", "Name": "original_email@test.com's workspace", "Path": "original_email@test.com's workspace", "Type": "Personal", "ParentKey": ""}, - {"Key": "a5b5c6d7-e8f9-0123-4567-89abcdef0123", "Name": "OpsAutomation", "Path": "OpsAutomation", "Description": "Folder hosting the OrderReconciliation REFramework process.", "Type": "Standard", "ParentKey": ""}, - {"Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", "Name": "Shared", "Path": "Shared", "Type": "Standard", "ParentKey": ""} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/manifest.json deleted file mode 100644 index 812c224645..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/m/r/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the max-consecutive-system-exceptions scenario (job-consecutive-system-exceptions playbook). The agent chains: folders list -> jobs list --state Faulted -> jobs get (Info: max consecutive system exceptions) -> jobs logs (the recurring SelectorNotFoundException on 'Click Submit', 5x, before the abort) -> jobs traces (same activity faults every transaction). Root cause = the recurring selector failure, not the threshold. Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key a5b5c6d7-e8f9-0123-4567-89abcdef0123 --state Faulted", "file": "12d7599762.json" }, - { "match": "or jobs get ccee6677-88ff-99aa-00bb-11cc22334455", "file": "aa692ef350.json" }, - { "match": "or jobs logs ccee6677-88ff-99aa-00bb-11cc22334455 --level Error", "file": "a812bb46c2.json" }, - { "match": "or jobs logs ccee6677-88ff-99aa-00bb-11cc22334455", "file": "0ea71af0c9.json" }, - { "match": "or jobs traces ccee6677-88ff-99aa-00bb-11cc22334455", "file": "6d148edb25.json" }, - { "match": "traces spans get --job-key ccee6677-88ff-99aa-00bb-11cc22334455", "file": "89dd6ac5f6.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0 } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/uip-fixture.json new file mode 100644 index 0000000000..a642b029c6 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/data/uip-fixture.json @@ -0,0 +1,91 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a5b5c6d7-e8f9-0123-4567-89abcdef0123", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"ccee6677-88ff-99aa-00bb-11cc22334455\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"OrderReconciliation\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-06-25T07:00:01.200Z\",\r\n \"EndTime\": \"2026-06-25T07:02:14.880Z\",\r\n \"CreationTime\": \"2026-06-25T07:00:01.000Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ccee6677-88ff-99aa-00bb-11cc22334455", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:02:14.870Z\", \"Message\": \"System exception. The maximum number of consecutive system exceptions was reached. Consecutive retry number: 5. Aborting the process.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:02:10.300Z\", \"Message\": \"Transaction 5 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:01:44.210Z\", \"Message\": \"Transaction 4 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:01:18.050Z\", \"Message\": \"Transaction 3 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:00:52.900Z\", \"Message\": \"Transaction 2 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:00:27.640Z\", \"Message\": \"Transaction 1 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "ccee6677-88ff-99aa-00bb-11cc22334455" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\"Activity\": \"Process Transaction (txn 1)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 2)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 3)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 4)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 5)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ccee6677-88ff-99aa-00bb-11cc22334455" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ccee6677-88ff-99aa-00bb-11cc22334455\",\r\n \"StartTime\": \"2026-06-25T07:00:01.200Z\",\r\n \"EndTime\": \"2026-06-25T07:02:14.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"The maximum number of consecutive system exceptions was reached. Consecutive retry number: 5.\",\r\n \"CreationTime\": \"2026-06-25T07:00:01.000Z\",\r\n \"ReleaseName\": \"OrderReconciliation\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"OrganizationUnitId\": 7791635,\r\n \"FolderKey\": \"a5b5c6d7-e8f9-0123-4567-89abcdef0123\",\r\n \"ProcessType\": \"Process\",\r\n \"LastModificationTime\": \"2026-06-25T07:02:14.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"ccee6677-88ff-99aa-00bb-11cc22334455\",\r\n \"Id\": 671660011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ccee6677-88ff-99aa-00bb-11cc22334455" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:02:14.870Z\", \"Message\": \"System exception. The maximum number of consecutive system exceptions was reached. Consecutive retry number: 5. Aborting the process.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:02:10.300Z\", \"Message\": \"Transaction 5 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:01:44.210Z\", \"Message\": \"Transaction 4 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:01:18.050Z\", \"Message\": \"Transaction 3 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:00:52.900Z\", \"Message\": \"Transaction 2 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-25T07:00:27.640Z\", \"Message\": \"Transaction 1 failed with a system exception: UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException: Could not find the UI element for 'Click Submit'. Selector: Timeout reached.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "ccee6677-88ff-99aa-00bb-11cc22334455" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\"Activity\": \"Process Transaction (txn 1)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 2)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 3)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 4)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}},\r\n {\"Activity\": \"Process Transaction (txn 5)\", \"Status\": \"Faulted\", \"Error\": {\"Type\": \"UiPath.UIAutomationNext.Exceptions.SelectorNotFoundException\", \"Activity\": \"Click Submit\", \"Message\": \"Could not find the UI element for 'Click Submit'.\"}}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\", \"Name\": \"original_email@test.com's workspace\", \"Path\": \"original_email@test.com's workspace\", \"Type\": \"Personal\", \"ParentKey\": \"\"},\r\n {\"Key\": \"a5b5c6d7-e8f9-0123-4567-89abcdef0123\", \"Name\": \"OpsAutomation\", \"Path\": \"OpsAutomation\", \"Description\": \"Folder hosting the OrderReconciliation REFramework process.\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\", \"Name\": \"Shared\", \"Path\": \"Shared\", \"Type\": \"Standard\", \"ParentKey\": \"\"}\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/task.yaml index 2e77e7997d..f88c39ece5 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-consecutive-system-exceptions/task.yaml @@ -26,21 +26,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/README.md new file mode 100644 index 0000000000..cdc543ad35 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/README.md @@ -0,0 +1,22 @@ +# Scenario: rpa-console-conflict-hd + +Synthetic Orchestrator control-plane scenario. Job `b2820001-2222-4222-8222-222222222222` +(`ClaimsIntake`, folder HD Automations) faults at executor start with +`Another interactive job is using the machine's console. Only one interactive job can use the console at a time.` +Maps to `products/orchestrator/playbooks/console-conflict-login-to-console.md`. + +The discriminating evidence: `machines list --all-fields` shows HD-BOT-POOL is a +high-density host (5 unattended slots, 4 robot users), and `jobs list` shows a +concurrent ClaimsIntake job Running on the same host. On an HD host, the console +message means the robots use **Login to Console = True** — all jobs attach to +the single physical console session, so only one interactive job runs at a time. +The fix is to disable Login to Console for the HD robots so each job gets its own +session — NOT a credential fix, a slot change, or a rerun. + +Control-plane failure (pre-workflow), so there is no `process/` source; the +agent diagnoses from `uip or jobs`/`machines` evidence. Grading: +`skill_triggered` + `llm_judge` vs `RESOLUTION.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/13914e1259.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/13914e1259.json deleted file mode 100644 index 7ddf279ff9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/13914e1259.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b2820001-2222-4222-8222-222222222222", - "StartTime": "2026-07-19T10:15:07.200Z", - "EndTime": "2026-07-19T10:15:08.050Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Queue", - "SourceType": "Queue", - "Info": "Another interactive job is using the machine's console. Only one interactive job can use the console at a time.\r\n at UiPath.Robot.Service.SessionManager.AttachToConsoleAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-07-19T10:15:06.600Z", - "ReleaseName": "ClaimsIntake", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": "HD-BOT-POOL", - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "FolderKey": "f2a20002-0000-4000-8000-000000000002", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\SVC_HD2", - "OrchestratorUserIdentity": "svc_hd2", - "LastModificationTime": "2026-07-19T10:15:08.050Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b2820001222242228222222222222222", - "Id": 671440202 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/1798ee29d5.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/1798ee29d5.json deleted file mode 100644 index c51101e87e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/1798ee29d5.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-19T10:15:08.050Z", - "JobId": 671440202 - }, - { - "State": "Running", - "Timestamp": "2026-07-19T10:15:07.210Z", - "JobId": 671440202 - }, - { - "State": "Pending", - "Timestamp": "2026-07-19T10:15:06.640Z", - "JobId": 671440202 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/18dc0563d3.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/18dc0563d3.json deleted file mode 100644 index 866c9a9f2e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/18dc0563d3.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T10:15:07.980Z", - "Message": "Another interactive job is using the machine's console. Only one interactive job can use the console at a time." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/75949d659e.json deleted file mode 100644 index afa8f76132..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/75949d659e.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "b2820001-2222-4222-8222-222222222222", - "State": "Faulted", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T10:15:07.200Z", - "EndTime": "2026-07-19T10:15:08.050Z", - "CreationTime": "2026-07-19T10:15:06.600Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - }, - { - "Key": "b2820002-3333-4333-8333-333333333333", - "State": "Running", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T10:15:05.900Z", - "EndTime": null, - "CreationTime": "2026-07-19T10:15:05.400Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - }, - { - "Key": "b2820003-4444-4444-8444-444444444444", - "State": "Successful", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T09:55:02.100Z", - "EndTime": "2026-07-19T09:58:41.700Z", - "CreationTime": "2026-07-19T09:55:01.600Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/8883fea2d6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/8883fea2d6.json deleted file mode 100644 index 9e63b292f0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/8883fea2d6.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "licenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2702", - "name": "HD-BOT-POOL", - "description": "High-density robot host (multiple robot users).", - "type": "Template", - "scope": "Default", - "nonProductionSlots": 0, - "unattendedSlots": 5, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "beef2001-2222-4333-8444-555566667702", - "endpointDetectionStatus": "Available", - "robotVersions": [ - { - "count": 4, - "version": "24.10.4" - } - ], - "robotUsers": [ - { "userName": "UIPATH\\SVC_HD1" }, - { "userName": "UIPATH\\SVC_HD2" }, - { "userName": "UIPATH\\SVC_HD3" }, - { "userName": "UIPATH\\SVC_HD4" } - ], - "automationType": "Any", - "targetFramework": "Windows", - "tags": [], - "id": 1990702 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/README.md deleted file mode 100644 index 513c4b6d52..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/README.md +++ /dev/null @@ -1,18 +0,0 @@ -# Scenario: rpa-console-conflict-hd - -Synthetic Orchestrator control-plane scenario. Job `b2820001-2222-4222-8222-222222222222` -(`ClaimsIntake`, folder HD Automations) faults at executor start with -`Another interactive job is using the machine's console. Only one interactive job can use the console at a time.` -Maps to `products/orchestrator/playbooks/console-conflict-login-to-console.md`. - -The discriminating evidence: `machines list --all-fields` shows HD-BOT-POOL is a -high-density host (5 unattended slots, 4 robot users), and `jobs list` shows a -concurrent ClaimsIntake job Running on the same host. On an HD host, the console -message means the robots use **Login to Console = True** — all jobs attach to -the single physical console session, so only one interactive job runs at a time. -The fix is to disable Login to Console for the HD robots so each job gets its own -session — NOT a credential fix, a slot change, or a rerun. - -Control-plane failure (pre-workflow), so there is no `process/` source; the -agent diagnoses from `uip or jobs`/`machines` evidence. Grading: -`skill_triggered` + `llm_judge` vs `RESOLUTION.md`. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/a189cd9315.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/a189cd9315.json deleted file mode 100644 index 3cdd8980e7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/a189cd9315.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "b2820001-2222-4222-8222-222222222222", - "State": "Faulted", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T10:15:07.200Z", - "EndTime": "2026-07-19T10:15:08.050Z", - "CreationTime": "2026-07-19T10:15:06.600Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/a760f714cb.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/a760f714cb.json deleted file mode 100644 index afa8f76132..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/a760f714cb.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "b2820001-2222-4222-8222-222222222222", - "State": "Faulted", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T10:15:07.200Z", - "EndTime": "2026-07-19T10:15:08.050Z", - "CreationTime": "2026-07-19T10:15:06.600Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - }, - { - "Key": "b2820002-3333-4333-8333-333333333333", - "State": "Running", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T10:15:05.900Z", - "EndTime": null, - "CreationTime": "2026-07-19T10:15:05.400Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - }, - { - "Key": "b2820003-4444-4444-8444-444444444444", - "State": "Successful", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T09:55:02.100Z", - "EndTime": "2026-07-19T09:58:41.700Z", - "CreationTime": "2026-07-19T09:55:01.600Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/d15031826a.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/d15031826a.json deleted file mode 100644 index 866c9a9f2e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/d15031826a.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T10:15:07.980Z", - "Message": "Another interactive job is using the machine's console. Only one interactive job can use the console at a time." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/dc064823f4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/dc064823f4.json deleted file mode 100644 index 99c06927b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/dc064823f4.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "name": "HD-BOT-POOL", - "description": "High-density robot host (multiple robot users).", - "type": "Template", - "key": "beef2001-2222-4333-8444-555566667702", - "id": 1990702 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/ee2628e6b6.json deleted file mode 100644 index 92f3b20775..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f2a20002-0000-4000-8000-000000000002", - "Name": "HD Automations", - "Path": "HD Automations", - "Description": "High-density robot pool automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/f5f2f217c8.json deleted file mode 100644 index 3cdd8980e7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "b2820001-2222-4222-8222-222222222222", - "State": "Faulted", - "ProcessName": "ClaimsIntake", - "ReleaseName": "ClaimsIntake", - "StartTime": "2026-07-19T10:15:07.200Z", - "EndTime": "2026-07-19T10:15:08.050Z", - "CreationTime": "2026-07-19T10:15:06.600Z", - "HostMachineName": "HD-BOT-POOL", - "Type": "Unattended", - "Source": "Queue", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/manifest.json deleted file mode 100644 index 3041acb561..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/m/r/manifest.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-console-conflict-hd scenario. The agent chains folders list -> jobs list --state Faulted -> jobs get -> jobs logs -> jobs history -> jobs list (concurrency) -> machines list --all-fields. Discriminating evidence: HD-BOT-POOL is a high-density host (5 unattended slots, 4 robot users), yet the job faulted with 'Another interactive job is using the machine's console' while a second ClaimsIntake job was Running on the same host — the single-console limit that Login to Console imposes on HD robots. Specific rules precede generic ones (first-match-wins).", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key f2a20002-0000-4000-8000-000000000002 --state Faulted", "file": "a189cd9315.json" }, - { "match": "or jobs list --folder-key f2a20002-0000-4000-8000-000000000002", "file": "a760f714cb.json" }, - { "match": "or jobs list --state Faulted", "file": "f5f2f217c8.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs get b2820001-2222-4222-8222-222222222222", "file": "13914e1259.json" }, - { "match": "or jobs logs b2820001-2222-4222-8222-222222222222 --level Error", "file": "d15031826a.json" }, - { "match": "or jobs logs b2820001-2222-4222-8222-222222222222", "file": "18dc0563d3.json" }, - { "match": "or jobs history b2820001-2222-4222-8222-222222222222", "file": "1798ee29d5.json" }, - { "match": "or machines list --all-fields", "file": "8883fea2d6.json" }, - { "match": "or machines list", "file": "dc064823f4.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/uip-fixture.json new file mode 100644 index 0000000000..5c66942f8b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/data/uip-fixture.json @@ -0,0 +1,134 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f2a20002-0000-4000-8000-000000000002", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2820001-2222-4222-8222-222222222222\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T10:15:07.200Z\",\r\n \"EndTime\": \"2026-07-19T10:15:08.050Z\",\r\n \"CreationTime\": \"2026-07-19T10:15:06.600Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b2820001-2222-4222-8222-222222222222", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T10:15:07.980Z\",\r\n \"Message\": \"Another interactive job is using the machine's console. Only one interactive job can use the console at a time.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f2a20002-0000-4000-8000-000000000002" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2820001-2222-4222-8222-222222222222\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T10:15:07.200Z\",\r\n \"EndTime\": \"2026-07-19T10:15:08.050Z\",\r\n \"CreationTime\": \"2026-07-19T10:15:06.600Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"b2820002-3333-4333-8333-333333333333\",\r\n \"State\": \"Running\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T10:15:05.900Z\",\r\n \"EndTime\": null,\r\n \"CreationTime\": \"2026-07-19T10:15:05.400Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"b2820003-4444-4444-8444-444444444444\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T09:55:02.100Z\",\r\n \"EndTime\": \"2026-07-19T09:58:41.700Z\",\r\n \"CreationTime\": \"2026-07-19T09:55:01.600Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2820001-2222-4222-8222-222222222222\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T10:15:07.200Z\",\r\n \"EndTime\": \"2026-07-19T10:15:08.050Z\",\r\n \"CreationTime\": \"2026-07-19T10:15:06.600Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b2820001-2222-4222-8222-222222222222" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b2820001-2222-4222-8222-222222222222\",\r\n \"StartTime\": \"2026-07-19T10:15:07.200Z\",\r\n \"EndTime\": \"2026-07-19T10:15:08.050Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Queue\",\r\n \"SourceType\": \"Queue\",\r\n \"Info\": \"Another interactive job is using the machine's console. Only one interactive job can use the console at a time.\\r\\n at UiPath.Robot.Service.SessionManager.AttachToConsoleAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-07-19T10:15:06.600Z\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"FolderKey\": \"f2a20002-0000-4000-8000-000000000002\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_HD2\",\r\n \"OrchestratorUserIdentity\": \"svc_hd2\",\r\n \"LastModificationTime\": \"2026-07-19T10:15:08.050Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b2820001222242228222222222222222\",\r\n \"Id\": 671440202\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b2820001-2222-4222-8222-222222222222" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T10:15:07.980Z\",\r\n \"Message\": \"Another interactive job is using the machine's console. Only one interactive job can use the console at a time.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "b2820001-2222-4222-8222-222222222222" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-19T10:15:08.050Z\",\r\n \"JobId\": 671440202\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-19T10:15:07.210Z\",\r\n \"JobId\": 671440202\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-19T10:15:06.640Z\",\r\n \"JobId\": 671440202\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--all-fields" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"licenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2702\",\r\n \"name\": \"HD-BOT-POOL\",\r\n \"description\": \"High-density robot host (multiple robot users).\",\r\n \"type\": \"Template\",\r\n \"scope\": \"Default\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 5,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"beef2001-2222-4333-8444-555566667702\",\r\n \"endpointDetectionStatus\": \"Available\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 4,\r\n \"version\": \"24.10.4\"\r\n }\r\n ],\r\n \"robotUsers\": [\r\n { \"userName\": \"UIPATH\\\\SVC_HD1\" },\r\n { \"userName\": \"UIPATH\\\\SVC_HD2\" },\r\n { \"userName\": \"UIPATH\\\\SVC_HD3\" },\r\n { \"userName\": \"UIPATH\\\\SVC_HD4\" }\r\n ],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Windows\",\r\n \"tags\": [],\r\n \"id\": 1990702\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f2a20002-0000-4000-8000-000000000002\",\r\n \"Name\": \"HD Automations\",\r\n \"Path\": \"HD Automations\",\r\n \"Description\": \"High-density robot pool automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2820001-2222-4222-8222-222222222222\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T10:15:07.200Z\",\r\n \"EndTime\": \"2026-07-19T10:15:08.050Z\",\r\n \"CreationTime\": \"2026-07-19T10:15:06.600Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"b2820002-3333-4333-8333-333333333333\",\r\n \"State\": \"Running\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T10:15:05.900Z\",\r\n \"EndTime\": null,\r\n \"CreationTime\": \"2026-07-19T10:15:05.400Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"b2820003-4444-4444-8444-444444444444\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"ClaimsIntake\",\r\n \"ReleaseName\": \"ClaimsIntake\",\r\n \"StartTime\": \"2026-07-19T09:55:02.100Z\",\r\n \"EndTime\": \"2026-07-19T09:58:41.700Z\",\r\n \"CreationTime\": \"2026-07-19T09:55:01.600Z\",\r\n \"HostMachineName\": \"HD-BOT-POOL\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Queue\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"name\": \"HD-BOT-POOL\",\r\n \"description\": \"High-density robot host (multiple robot users).\",\r\n \"type\": \"Template\",\r\n \"key\": \"beef2001-2222-4333-8444-555566667702\",\r\n \"id\": 1990702\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/task.yaml index 124412f971..1a231140db 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-console-conflict-hd/task.yaml @@ -17,20 +17,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/README.md new file mode 100644 index 0000000000..8dbfeaf103 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/README.md @@ -0,0 +1,53 @@ +# CredProvider Path — Known Robot Defect (UiPath KB 799589) + +Reproduces the `known-issue-robot-defect` playbook: a `Could not +start executor` fault whose signature matches a documented, +already-fixed Robot defect. + +``` +Could not start executor. Could not find a part of the path +'C:\Windows\TEMP\UiPath\CredProvider'. +``` + +## What this scenario uncovers + +**Root Cause:** A known Robot defect (UiPath KB 799589) fixed in +Robot 23.4.11 / 23.10.10 / 24.10.7. The host runs 23.10.4 (predates +the fix). The correct fix is to **upgrade the Robot to ≥ 23.10.10** — +not to hand-create the missing folder or change TEMP permissions. + +Maps to: +`references/products/orchestrator/playbooks/known-issue-robot-defect.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal unattended UiPath project | + +> Fixtures authored from the playbook signature (and the UiPath KB +> 799589 fix-version fact), not captured from a real session. + +## Distinguishing fingerprint / common wrong turn + +The Info names a missing TEMP path, which tempts a naive "create the +folder / fix TEMP permissions" answer. The playbook-driven diagnosis +recognizes the signature as a **known Robot defect** and checks the +host Robot version (23.10.4 < 23.10.10) — the correct fix is a Robot +upgrade. An agent that suggests recreating the directory scores low; +one that identifies the known issue and recommends the upgrade scores +full. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the CredProvider-path fault as a known Robot + defect (fixed in 23.10.10 on the 23.10 line) given the host's older + 23.10.4 build, and recommended upgrading the Robot to ≥ 23.10.10 + rather than hand-creating the folder or editing TEMP permissions. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/3d6f8cbb3b.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/3d6f8cbb3b.json deleted file mode 100644 index 70c2ea357c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/3d6f8cbb3b.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bbccddee-2233-4455-6677-8899aabbccdd", - "StartTime": "2026-06-28T06:00:01.400Z", - "EndTime": "2026-06-28T06:00:02.010Z", - "State": "Faulted", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Could not start executor. Could not find a part of the path 'C:\\Windows\\TEMP\\UiPath\\CredProvider'.", - "CreationTime": "2026-06-28T06:00:01.300Z", - "ReleaseName": "LedgerPostingBot", - "Type": "Unattended", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "OrganizationUnitId": 7791645, - "FolderKey": "f6d6e7f8-a9b0-1234-5678-9abcdef01234", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\SVCLEDGER", - "MachineName": "MOCK-HOST", - "MachineKey": "c3d4e5f6-a7b8-9012-3456-789abcdef012", - "LastModificationTime": "2026-06-28T06:00:02.010Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "bbccddee-2233-4455-6677-8899aabbccdd", - "Id": 671990011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/7f45d08f0f.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/7f45d08f0f.json deleted file mode 100644 index a8aa1c37af..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/7f45d08f0f.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-28T06:00:02.000Z", "Message": "Job 'bbccddee-2233-4455-6677-8899aabbccdd' failed to start. Could not start executor. Could not find a part of the path 'C:\\Windows\\TEMP\\UiPath\\CredProvider'."}, - {"Level": "Error", "TimeStamp": "2026-06-28T06:00:01.850Z", "Message": "[Robot] Executor start failed: the credential-provider log directory 'C:\\Windows\\TEMP\\UiPath\\CredProvider' is missing."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/README.md deleted file mode 100644 index d28bd27eb2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/README.md +++ /dev/null @@ -1,52 +0,0 @@ -# CredProvider Path — Known Robot Defect (UiPath KB 799589) - -Reproduces the `known-issue-robot-defect` playbook: a `Could not -start executor` fault whose signature matches a documented, -already-fixed Robot defect. - -``` -Could not start executor. Could not find a part of the path -'C:\Windows\TEMP\UiPath\CredProvider'. -``` - -## What this scenario uncovers - -**Root Cause:** A known Robot defect (UiPath KB 799589) fixed in -Robot 23.4.11 / 23.10.10 / 24.10.7. The host runs 23.10.4 (predates -the fix). The correct fix is to **upgrade the Robot to ≥ 23.10.10** — -not to hand-create the missing folder or change TEMP permissions. - -Maps to: -`references/products/orchestrator/playbooks/known-issue-robot-defect.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal unattended UiPath project | -| `data/m/r/*.json` | **synthetic** canned `uip` responses — jobs get/list/logs plus `machines list` showing Robot 23.10.4 | -| `data/m/r/manifest.json` | dispatch table | - -> Fixtures authored from the playbook signature (and the UiPath KB -> 799589 fix-version fact), not captured from a real session. - -## Distinguishing fingerprint / common wrong turn - -The Info names a missing TEMP path, which tempts a naive "create the -folder / fix TEMP permissions" answer. The playbook-driven diagnosis -recognizes the signature as a **known Robot defect** and checks the -host Robot version (23.10.4 < 23.10.10) — the correct fix is a Robot -upgrade. An agent that suggests recreating the directory scores low; -one that identifies the known issue and recommends the upgrade scores -full. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the CredProvider-path fault as a known Robot - defect (fixed in 23.10.10 on the 23.10 line) given the host's older - 23.10.4 build, and recommended upgrading the Robot to ≥ 23.10.10 - rather than hand-creating the folder or editing TEMP permissions. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/a8569d7fc7.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/a8569d7fc7.json deleted file mode 100644 index 8cf4160a36..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/a8569d7fc7.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - {"Key": "bbccddee-2233-4455-6677-8899aabbccdd", "State": "Faulted", "ProcessName": "LedgerPostingBot", "Source": "Schedule", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST", "StartTime": "2026-06-28T06:00:01.400Z", "EndTime": "2026-06-28T06:00:02.010Z", "CreationTime": "2026-06-28T06:00:01.300Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/dc064823f4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/dc064823f4.json deleted file mode 100644 index d514ca923b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/dc064823f4.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - {"Key": "c3d4e5f6-a7b8-9012-3456-789abcdef012", "Name": "BackOfficeRuntime", "Type": "Template", "Runtimes": 2, "Hosts": ["MOCK-HOST"], "RobotVersion": "23.10.4", "Description": "Runtime template for BackOffice; hosts run Robot 23.10.4."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/ee2628e6b6.json deleted file mode 100644 index 5064cc8404..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - {"Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", "Name": "original_email@test.com's workspace", "Path": "original_email@test.com's workspace", "Type": "Personal", "ParentKey": ""}, - {"Key": "f6d6e7f8-a9b0-1234-5678-9abcdef01234", "Name": "BackOffice", "Path": "BackOffice", "Description": "Folder hosting the LedgerPostingBot unattended process.", "Type": "Standard", "ParentKey": ""}, - {"Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", "Name": "Shared", "Path": "Shared", "Type": "Standard", "ParentKey": ""} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/fb379d10d9.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/fb379d10d9.json deleted file mode 100644 index a8aa1c37af..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/fb379d10d9.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-28T06:00:02.000Z", "Message": "Job 'bbccddee-2233-4455-6677-8899aabbccdd' failed to start. Could not start executor. Could not find a part of the path 'C:\\Windows\\TEMP\\UiPath\\CredProvider'."}, - {"Level": "Error", "TimeStamp": "2026-06-28T06:00:01.850Z", "Message": "[Robot] Executor start failed: the credential-provider log directory 'C:\\Windows\\TEMP\\UiPath\\CredProvider' is missing."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/manifest.json deleted file mode 100644 index 4d20e18504..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/m/r/manifest.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the CredProvider-path known-issue scenario (known-issue-robot-defect playbook). The agent chains: folders list -> jobs list --state Faulted -> jobs get (Info: CredProvider TEMP path) -> jobs logs (known defect on builds <= 23.10.8) -> machines list (RobotVersion 23.10.4). Correct fix: upgrade Robot to the fix version (23.4.11 / 23.10.10 / 24.10.7, UiPath KB 799589), NOT create the folder / fix TEMP permissions. Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key f6d6e7f8-a9b0-1234-5678-9abcdef01234 --state Faulted", "file": "a8569d7fc7.json" }, - { "match": "or jobs get bbccddee-2233-4455-6677-8899aabbccdd", "file": "3d6f8cbb3b.json" }, - { "match": "or jobs logs bbccddee-2233-4455-6677-8899aabbccdd --level Error", "file": "7f45d08f0f.json" }, - { "match": "or jobs logs bbccddee-2233-4455-6677-8899aabbccdd", "file": "fb379d10d9.json" }, - { "match": "or machines list", "file": "dc064823f4.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0 } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/uip-fixture.json new file mode 100644 index 0000000000..d43bf85f5b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/data/uip-fixture.json @@ -0,0 +1,78 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f6d6e7f8-a9b0-1234-5678-9abcdef01234", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\"Key\": \"bbccddee-2233-4455-6677-8899aabbccdd\", \"State\": \"Faulted\", \"ProcessName\": \"LedgerPostingBot\", \"Source\": \"Schedule\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST\", \"StartTime\": \"2026-06-28T06:00:01.400Z\", \"EndTime\": \"2026-06-28T06:00:02.010Z\", \"CreationTime\": \"2026-06-28T06:00:01.300Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bbccddee-2233-4455-6677-8899aabbccdd", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-28T06:00:02.000Z\", \"Message\": \"Job 'bbccddee-2233-4455-6677-8899aabbccdd' failed to start. Could not start executor. Could not find a part of the path 'C:\\\\Windows\\\\TEMP\\\\UiPath\\\\CredProvider'.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-28T06:00:01.850Z\", \"Message\": \"[Robot] Executor start failed: the credential-provider log directory 'C:\\\\Windows\\\\TEMP\\\\UiPath\\\\CredProvider' is missing.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bbccddee-2233-4455-6677-8899aabbccdd" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bbccddee-2233-4455-6677-8899aabbccdd\",\r\n \"StartTime\": \"2026-06-28T06:00:01.400Z\",\r\n \"EndTime\": \"2026-06-28T06:00:02.010Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Could not start executor. Could not find a part of the path 'C:\\\\Windows\\\\TEMP\\\\UiPath\\\\CredProvider'.\",\r\n \"CreationTime\": \"2026-06-28T06:00:01.300Z\",\r\n \"ReleaseName\": \"LedgerPostingBot\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"OrganizationUnitId\": 7791645,\r\n \"FolderKey\": \"f6d6e7f8-a9b0-1234-5678-9abcdef01234\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVCLEDGER\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"MachineKey\": \"c3d4e5f6-a7b8-9012-3456-789abcdef012\",\r\n \"LastModificationTime\": \"2026-06-28T06:00:02.010Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bbccddee-2233-4455-6677-8899aabbccdd\",\r\n \"Id\": 671990011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bbccddee-2233-4455-6677-8899aabbccdd" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-28T06:00:02.000Z\", \"Message\": \"Job 'bbccddee-2233-4455-6677-8899aabbccdd' failed to start. Could not start executor. Could not find a part of the path 'C:\\\\Windows\\\\TEMP\\\\UiPath\\\\CredProvider'.\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-28T06:00:01.850Z\", \"Message\": \"[Robot] Executor start failed: the credential-provider log directory 'C:\\\\Windows\\\\TEMP\\\\UiPath\\\\CredProvider' is missing.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\", \"Name\": \"original_email@test.com's workspace\", \"Path\": \"original_email@test.com's workspace\", \"Type\": \"Personal\", \"ParentKey\": \"\"},\r\n {\"Key\": \"f6d6e7f8-a9b0-1234-5678-9abcdef01234\", \"Name\": \"BackOffice\", \"Path\": \"BackOffice\", \"Description\": \"Folder hosting the LedgerPostingBot unattended process.\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\", \"Name\": \"Shared\", \"Path\": \"Shared\", \"Type\": \"Standard\", \"ParentKey\": \"\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\"Key\": \"c3d4e5f6-a7b8-9012-3456-789abcdef012\", \"Name\": \"BackOfficeRuntime\", \"Type\": \"Template\", \"Runtimes\": 2, \"Hosts\": [\"MOCK-HOST\"], \"RobotVersion\": \"23.10.4\", \"Description\": \"Runtime template for BackOffice; hosts run Robot 23.10.4.\"}\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/task.yaml index 631995b3a3..ee7059ad83 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-credprovider-path-known-issue/task.yaml @@ -26,21 +26,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/README.md new file mode 100644 index 0000000000..95f65e5d74 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/README.md @@ -0,0 +1,21 @@ +# Scenario: rpa-executor-service-disconnect + +Synthetic Orchestrator control-plane scenario. Job `d4a40001-4444-4444-8444-444444444444` +(`NightlyPost`, folder Batch Ops) ran ~68s on BATCH-BOT-02 then faulted with +`Job faulted due to service shutdown or disconnect.` Maps to +`products/orchestrator/playbooks/executor-start-transient-rerun.md` (rerun-class). + +The discriminating evidence: the message names a **service shutdown/disconnect** +(not a logon code, session-timeout, slot/console, or quota), history is +Pending → Running → Faulted, and `jobs list` shows the **previous nights' +NightlyPost runs on the same host succeeded** — an intermittent transient. The +fix is to **rerun** (escalating to host/robot stability + Robot update only if +it recurs) — NOT a credential, slot, license, or code change. + +Control-plane failure, so there is no `process/` source; the agent diagnoses +from `uip or jobs`/`machines` evidence. Grading: `skill_triggered` + +`llm_judge` vs `RESOLUTION.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/04fb2ee096.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/04fb2ee096.json deleted file mode 100644 index d615d775c6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/04fb2ee096.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "d4a40001-4444-4444-8444-444444444444", - "State": "Faulted", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-19T01:30:04.200Z", - "EndTime": "2026-07-19T01:31:12.600Z", - "CreationTime": "2026-07-19T01:30:03.700Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "d4a40002-5555-4555-8555-555555555555", - "State": "Successful", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-18T01:30:03.900Z", - "EndTime": "2026-07-18T01:52:20.400Z", - "CreationTime": "2026-07-18T01:30:03.400Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "d4a40003-6666-4666-8666-666666666666", - "State": "Successful", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-17T01:30:04.100Z", - "EndTime": "2026-07-17T01:49:58.900Z", - "CreationTime": "2026-07-17T01:30:03.600Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/24be87781f.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/24be87781f.json deleted file mode 100644 index 303a910cb0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/24be87781f.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "d4a40001-4444-4444-8444-444444444444", - "State": "Faulted", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-19T01:30:04.200Z", - "EndTime": "2026-07-19T01:31:12.600Z", - "CreationTime": "2026-07-19T01:30:03.700Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/75949d659e.json deleted file mode 100644 index d615d775c6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/75949d659e.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "d4a40001-4444-4444-8444-444444444444", - "State": "Faulted", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-19T01:30:04.200Z", - "EndTime": "2026-07-19T01:31:12.600Z", - "CreationTime": "2026-07-19T01:30:03.700Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "d4a40002-5555-4555-8555-555555555555", - "State": "Successful", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-18T01:30:03.900Z", - "EndTime": "2026-07-18T01:52:20.400Z", - "CreationTime": "2026-07-18T01:30:03.400Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "d4a40003-6666-4666-8666-666666666666", - "State": "Successful", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-17T01:30:04.100Z", - "EndTime": "2026-07-17T01:49:58.900Z", - "CreationTime": "2026-07-17T01:30:03.600Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/8883fea2d6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/8883fea2d6.json deleted file mode 100644 index 0571928fd5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/8883fea2d6.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "licenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2704", - "name": "BATCH-BOT-02", - "description": "Overnight batch unattended host.", - "type": "Template", - "scope": "Default", - "nonProductionSlots": 0, - "unattendedSlots": 1, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "beef4001-2222-4333-8444-555566667704", - "endpointDetectionStatus": "Available", - "robotVersions": [ - { - "count": 1, - "version": "24.10.4" - } - ], - "robotUsers": [ - { "userName": "UIPATH\\SVC_BATCH" } - ], - "automationType": "Any", - "targetFramework": "Windows", - "tags": [], - "id": 1990704 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/9528d0e2e6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/9528d0e2e6.json deleted file mode 100644 index d734fa6a04..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/9528d0e2e6.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T01:31:12.500Z", - "Message": "Job faulted due to service shutdown or disconnect." - }, - { - "Level": "Info", - "TimeStamp": "2026-07-19T01:30:20.100Z", - "Message": "NightlyPost execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/9fc2c63365.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/9fc2c63365.json deleted file mode 100644 index d734fa6a04..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/9fc2c63365.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T01:31:12.500Z", - "Message": "Job faulted due to service shutdown or disconnect." - }, - { - "Level": "Info", - "TimeStamp": "2026-07-19T01:30:20.100Z", - "Message": "NightlyPost execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/README.md deleted file mode 100644 index 5eda1d0746..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Scenario: rpa-executor-service-disconnect - -Synthetic Orchestrator control-plane scenario. Job `d4a40001-4444-4444-8444-444444444444` -(`NightlyPost`, folder Batch Ops) ran ~68s on BATCH-BOT-02 then faulted with -`Job faulted due to service shutdown or disconnect.` Maps to -`products/orchestrator/playbooks/executor-start-transient-rerun.md` (rerun-class). - -The discriminating evidence: the message names a **service shutdown/disconnect** -(not a logon code, session-timeout, slot/console, or quota), history is -Pending → Running → Faulted, and `jobs list` shows the **previous nights' -NightlyPost runs on the same host succeeded** — an intermittent transient. The -fix is to **rerun** (escalating to host/robot stability + Robot update only if -it recurs) — NOT a credential, slot, license, or code change. - -Control-plane failure, so there is no `process/` source; the agent diagnoses -from `uip or jobs`/`machines` evidence. Grading: `skill_triggered` + -`llm_judge` vs `RESOLUTION.md`. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/ca4c96b7c6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/ca4c96b7c6.json deleted file mode 100644 index e593b3e2f5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/ca4c96b7c6.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-19T01:31:12.600Z", - "JobId": 671660404 - }, - { - "State": "Running", - "Timestamp": "2026-07-19T01:30:04.210Z", - "JobId": 671660404 - }, - { - "State": "Pending", - "Timestamp": "2026-07-19T01:30:03.740Z", - "JobId": 671660404 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/dc064823f4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/dc064823f4.json deleted file mode 100644 index 9d6b7a956d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/dc064823f4.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "name": "BATCH-BOT-02", - "description": "Overnight batch unattended host.", - "type": "Template", - "key": "beef4001-2222-4333-8444-555566667704", - "id": 1990704 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/dc90f8e6eb.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/dc90f8e6eb.json deleted file mode 100644 index 979f536726..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/dc90f8e6eb.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d4a40001-4444-4444-8444-444444444444", - "StartTime": "2026-07-19T01:30:04.200Z", - "EndTime": "2026-07-19T01:31:12.600Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Job faulted due to service shutdown or disconnect.", - "CreationTime": "2026-07-19T01:30:03.700Z", - "ReleaseName": "NightlyPost", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": "BATCH-BOT-02", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "FolderKey": "f4a40004-0000-4000-8000-000000000004", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\SVC_BATCH", - "OrchestratorUserIdentity": "svc_batch", - "LastModificationTime": "2026-07-19T01:31:12.600Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d4a40001444444448444444444444444", - "Id": 671660404 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/ee2628e6b6.json deleted file mode 100644 index 2957e0574a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f4a40004-0000-4000-8000-000000000004", - "Name": "Batch Ops", - "Path": "Batch Ops", - "Description": "Overnight batch automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/f5f2f217c8.json deleted file mode 100644 index 303a910cb0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "d4a40001-4444-4444-8444-444444444444", - "State": "Faulted", - "ProcessName": "NightlyPost", - "ReleaseName": "NightlyPost", - "StartTime": "2026-07-19T01:30:04.200Z", - "EndTime": "2026-07-19T01:31:12.600Z", - "CreationTime": "2026-07-19T01:30:03.700Z", - "HostMachineName": "BATCH-BOT-02", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/manifest.json deleted file mode 100644 index a029acc5fc..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/m/r/manifest.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-executor-service-disconnect scenario. The agent chains folders list -> jobs list --state Faulted -> jobs get -> jobs logs -> jobs history -> jobs list (intermittency) -> machines list. Discriminating evidence: the message is 'Job faulted due to service shutdown or disconnect' (a transient robot-service event, NOT a logon code / session-timeout / slot / console / quota), history is Pending->Running->Faulted after ~68s of runtime, and prior NightlyPost runs on the same host BATCH-BOT-02 succeeded — an intermittent transient. Shared fix: rerun (rerun-class playbook). Specific rules precede generic ones (first-match-wins).", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key f4a40004-0000-4000-8000-000000000004 --state Faulted", "file": "24be87781f.json" }, - { "match": "or jobs list --folder-key f4a40004-0000-4000-8000-000000000004", "file": "04fb2ee096.json" }, - { "match": "or jobs list --state Faulted", "file": "f5f2f217c8.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs get d4a40001-4444-4444-8444-444444444444", "file": "dc90f8e6eb.json" }, - { "match": "or jobs logs d4a40001-4444-4444-8444-444444444444 --level Error", "file": "9528d0e2e6.json" }, - { "match": "or jobs logs d4a40001-4444-4444-8444-444444444444", "file": "9fc2c63365.json" }, - { "match": "or jobs history d4a40001-4444-4444-8444-444444444444", "file": "ca4c96b7c6.json" }, - { "match": "or machines list --all-fields", "file": "8883fea2d6.json" }, - { "match": "or machines list", "file": "dc064823f4.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/uip-fixture.json new file mode 100644 index 0000000000..2f38e70a67 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/data/uip-fixture.json @@ -0,0 +1,134 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f4a40004-0000-4000-8000-000000000004", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4a40001-4444-4444-8444-444444444444\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-19T01:30:04.200Z\",\r\n \"EndTime\": \"2026-07-19T01:31:12.600Z\",\r\n \"CreationTime\": \"2026-07-19T01:30:03.700Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d4a40001-4444-4444-8444-444444444444", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T01:31:12.500Z\",\r\n \"Message\": \"Job faulted due to service shutdown or disconnect.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-19T01:30:20.100Z\",\r\n \"Message\": \"NightlyPost execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f4a40004-0000-4000-8000-000000000004" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4a40001-4444-4444-8444-444444444444\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-19T01:30:04.200Z\",\r\n \"EndTime\": \"2026-07-19T01:31:12.600Z\",\r\n \"CreationTime\": \"2026-07-19T01:30:03.700Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"d4a40002-5555-4555-8555-555555555555\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-18T01:30:03.900Z\",\r\n \"EndTime\": \"2026-07-18T01:52:20.400Z\",\r\n \"CreationTime\": \"2026-07-18T01:30:03.400Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"d4a40003-6666-4666-8666-666666666666\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-17T01:30:04.100Z\",\r\n \"EndTime\": \"2026-07-17T01:49:58.900Z\",\r\n \"CreationTime\": \"2026-07-17T01:30:03.600Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4a40001-4444-4444-8444-444444444444\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-19T01:30:04.200Z\",\r\n \"EndTime\": \"2026-07-19T01:31:12.600Z\",\r\n \"CreationTime\": \"2026-07-19T01:30:03.700Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d4a40001-4444-4444-8444-444444444444" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d4a40001-4444-4444-8444-444444444444\",\r\n \"StartTime\": \"2026-07-19T01:30:04.200Z\",\r\n \"EndTime\": \"2026-07-19T01:31:12.600Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Job faulted due to service shutdown or disconnect.\",\r\n \"CreationTime\": \"2026-07-19T01:30:03.700Z\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"FolderKey\": \"f4a40004-0000-4000-8000-000000000004\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_BATCH\",\r\n \"OrchestratorUserIdentity\": \"svc_batch\",\r\n \"LastModificationTime\": \"2026-07-19T01:31:12.600Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d4a40001444444448444444444444444\",\r\n \"Id\": 671660404\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d4a40001-4444-4444-8444-444444444444" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T01:31:12.500Z\",\r\n \"Message\": \"Job faulted due to service shutdown or disconnect.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-07-19T01:30:20.100Z\",\r\n \"Message\": \"NightlyPost execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "d4a40001-4444-4444-8444-444444444444" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-19T01:31:12.600Z\",\r\n \"JobId\": 671660404\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-19T01:30:04.210Z\",\r\n \"JobId\": 671660404\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-19T01:30:03.740Z\",\r\n \"JobId\": 671660404\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--all-fields" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"licenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2704\",\r\n \"name\": \"BATCH-BOT-02\",\r\n \"description\": \"Overnight batch unattended host.\",\r\n \"type\": \"Template\",\r\n \"scope\": \"Default\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 1,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"beef4001-2222-4333-8444-555566667704\",\r\n \"endpointDetectionStatus\": \"Available\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"24.10.4\"\r\n }\r\n ],\r\n \"robotUsers\": [\r\n { \"userName\": \"UIPATH\\\\SVC_BATCH\" }\r\n ],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Windows\",\r\n \"tags\": [],\r\n \"id\": 1990704\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f4a40004-0000-4000-8000-000000000004\",\r\n \"Name\": \"Batch Ops\",\r\n \"Path\": \"Batch Ops\",\r\n \"Description\": \"Overnight batch automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d4a40001-4444-4444-8444-444444444444\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-19T01:30:04.200Z\",\r\n \"EndTime\": \"2026-07-19T01:31:12.600Z\",\r\n \"CreationTime\": \"2026-07-19T01:30:03.700Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"d4a40002-5555-4555-8555-555555555555\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-18T01:30:03.900Z\",\r\n \"EndTime\": \"2026-07-18T01:52:20.400Z\",\r\n \"CreationTime\": \"2026-07-18T01:30:03.400Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"d4a40003-6666-4666-8666-666666666666\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"NightlyPost\",\r\n \"ReleaseName\": \"NightlyPost\",\r\n \"StartTime\": \"2026-07-17T01:30:04.100Z\",\r\n \"EndTime\": \"2026-07-17T01:49:58.900Z\",\r\n \"CreationTime\": \"2026-07-17T01:30:03.600Z\",\r\n \"HostMachineName\": \"BATCH-BOT-02\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"name\": \"BATCH-BOT-02\",\r\n \"description\": \"Overnight batch unattended host.\",\r\n \"type\": \"Template\",\r\n \"key\": \"beef4001-2222-4333-8444-555566667704\",\r\n \"id\": 1990704\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/task.yaml index da33e03f24..adb4fcedcb 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-executor-service-disconnect/task.yaml @@ -17,20 +17,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/README.md new file mode 100644 index 0000000000..167d76cbc4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/README.md @@ -0,0 +1,79 @@ +# Rpa Foreground Already Running — Concurrent Foreground Job Rejection + +This scenario reproduces a runtime fault caused by **two foreground +(UI-interactive) jobs scheduled against the same Robot session with +overlapping start times**. The Robot rejects the second start with: + +``` +System.InvalidOperationException: A foreground process is already running. +Only one foreground process can run at a time. +``` + +## What this scenario uncovers + +**Root Cause:** A second `AttendedReportJob` job (key `c0a1b2c3-...`) +was triggered at `2026-05-12T10:15:00Z` while an earlier +`AttendedReportJob` job (key `b0a1b2c3-...`, started +`2026-05-12T10:14:50Z`) was still in `Running` state on the same +machine. Both have `RequiresUserInteraction: true` (foreground). The +Robot enforces a single-foreground-job-per-session constraint and +faults the second start within ~0.7s. + +This maps to: +`references/products/orchestrator/playbooks/foreground-already-running.md` +(the "Two foreground triggers fire on overlapping schedules" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of `skills/uipath-troubleshoot/fixtures/foreground-already-running/AttendedReportJob/` — the foreground project that's failing | + +> **Note on fixtures.** Like sibling synthetic scenarios, the fixtures +> here were authored from the documented playbook signature rather +> than captured from a real `.investigation/` session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression signal. + +## How this differs from the sibling foreground playbook + +There are **two** UiPath playbooks that mention "foreground" — agents +must pick the right one: + +| Dimension | `maestro/foreground-unattended-robot` (sibling) | `orchestrator/foreground-already-running` (this) | +|---|---|---| +| HTTP status | 409 | n/a (Robot-side guard, not HTTP) | +| Error code / class | Orchestrator `#1230` | .NET `System.InvalidOperationException` | +| Error message anchor | "Foreground job requires an unattended robot to be defined on your user" | "A foreground process is already running. Only one foreground process can run at a time." | +| Root cause class | Missing unattended robot configuration on the user | Concurrent foreground execution on a single Robot session | +| Fix | Provision unattended robot + machine credentials | Sequence triggers / "Run only one job at a time" / convert to background | + +A correct agent matches the **this** playbook by the exception text +and the presence of an overlapping Running job in the same folder, +NOT by the word "foreground" alone. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `orchestrator/foreground-already-running` (not the + maestro `#1230` playbook) AND reached the same root cause as + `RESOLUTION.md` +- Conclusion must explicitly name the concurrent foreground job + conflict (overlapping Running job at the time of failure) and + propose a valid fix from the playbook's resolution list + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name rpa-foreground-already-running --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/1e29694194.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/1e29694194.json deleted file mode 100644 index 21948a02f7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/1e29694194.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T10:15:01.215Z", - "Message": "Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\r\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T10:15:01.190Z", - "Message": "[Robot] Cannot start foreground job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' (ReleaseName=AttendedReportJob). Another foreground job ('b0a1b2c3-d4e5-4678-9012-3456789abcde', ReleaseName=AttendedReportJob, started 2026-05-12T10:14:50.000Z) is currently running on this session. Only one foreground process can run at a time." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/3603e33798.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/3603e33798.json deleted file mode 100644 index c82d751455..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/3603e33798.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent jobs in ForegroundDemo. The failing job (c0a1b2c3) was created while the earlier AttendedReportJob run (b0a1b2c3) was still Running — overlap is the key evidence for the foreground-already-running playbook.", - "Data": [ - { - "Key": "c0a1b2c3-d4e5-4678-9012-3456789abcde", - "State": "Faulted", - "ProcessName": "AttendedReportJob", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T10:15:00.500Z", - "EndTime": "2026-05-12T10:15:01.218Z", - "CreationTime": "2026-05-12T10:15:00.302Z" - }, - { - "Key": "b0a1b2c3-d4e5-4678-9012-3456789abcde", - "State": "Successful", - "ProcessName": "AttendedReportJob", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T10:14:50.000Z", - "EndTime": "2026-05-12T10:15:50.412Z", - "CreationTime": "2026-05-12T10:14:49.852Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/README.md deleted file mode 100644 index 0220aa138a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/README.md +++ /dev/null @@ -1,78 +0,0 @@ -# Rpa Foreground Already Running — Concurrent Foreground Job Rejection - -This scenario reproduces a runtime fault caused by **two foreground -(UI-interactive) jobs scheduled against the same Robot session with -overlapping start times**. The Robot rejects the second start with: - -``` -System.InvalidOperationException: A foreground process is already running. -Only one foreground process can run at a time. -``` - -## What this scenario uncovers - -**Root Cause:** A second `AttendedReportJob` job (key `c0a1b2c3-...`) -was triggered at `2026-05-12T10:15:00Z` while an earlier -`AttendedReportJob` job (key `b0a1b2c3-...`, started -`2026-05-12T10:14:50Z`) was still in `Running` state on the same -machine. Both have `RequiresUserInteraction: true` (foreground). The -Robot enforces a single-foreground-job-per-session constraint and -faults the second start within ~0.7s. - -This maps to: -`references/products/orchestrator/playbooks/foreground-already-running.md` -(the "Two foreground triggers fire on overlapping schedules" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | snapshot of `skills/uipath-troubleshoot/fixtures/foreground-already-running/AttendedReportJob/` — the foreground project that's failing | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like sibling synthetic scenarios, the fixtures -> here were authored from the documented playbook signature rather -> than captured from a real `.investigation/` session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression signal. - -## How this differs from the sibling foreground playbook - -There are **two** UiPath playbooks that mention "foreground" — agents -must pick the right one: - -| Dimension | `maestro/foreground-unattended-robot` (sibling) | `orchestrator/foreground-already-running` (this) | -|---|---|---| -| HTTP status | 409 | n/a (Robot-side guard, not HTTP) | -| Error code / class | Orchestrator `#1230` | .NET `System.InvalidOperationException` | -| Error message anchor | "Foreground job requires an unattended robot to be defined on your user" | "A foreground process is already running. Only one foreground process can run at a time." | -| Root cause class | Missing unattended robot configuration on the user | Concurrent foreground execution on a single Robot session | -| Fix | Provision unattended robot + machine credentials | Sequence triggers / "Run only one job at a time" / convert to background | - -A correct agent matches the **this** playbook by the exception text -and the presence of an overlapping Running job in the same folder, -NOT by the word "foreground" alone. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `orchestrator/foreground-already-running` (not the - maestro `#1230` playbook) AND reached the same root cause as - `RESOLUTION.md` -- Conclusion must explicitly name the concurrent foreground job - conflict (overlapping Running job at the time of failure) and - propose a valid fix from the playbook's resolution list - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name rpa-foreground-already-running --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/c8f5871c1b.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/c8f5871c1b.json deleted file mode 100644 index 986c86f542..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/c8f5871c1b.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c0a1b2c3-d4e5-4678-9012-3456789abcde", - "StartTime": "2026-05-12T10:15:00.500Z", - "EndTime": "2026-05-12T10:15:01.218Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "9d2e7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f", - "Info": "System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\r\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-12T10:15:00.302Z", - "ReleaseName": "AttendedReportJob", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 4670012, - "OrganizationUnitId": 7791617, - "FolderKey": "a1b2c3d4-e5f6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T10:15:01.218Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c0a1b2c3-d4e5-4678-9012-3456789abcde", - "Id": 670910207 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/dd51f4fc27.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/dd51f4fc27.json deleted file mode 100644 index 254a39a0b7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/dd51f4fc27.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-12T10:15:01.215Z", - "Message": "Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\r\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-12T10:15:01.190Z", - "Message": "[Robot] Cannot start foreground job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' (ReleaseName=AttendedReportJob). Another foreground job ('b0a1b2c3-d4e5-4678-9012-3456789abcde', ReleaseName=AttendedReportJob, started 2026-05-12T10:14:50.000Z) is currently running on this session. Only one foreground process can run at a time." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T10:15:00.510Z", - "Message": "Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' (ReleaseName=AttendedReportJob) entering Running state on MOCK-HOST." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-12T10:15:00.310Z", - "Message": "Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' created. Source=Manual, ReleaseName=AttendedReportJob, RequiresUserInteraction=True." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/df6832044b.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/df6832044b.json deleted file mode 100644 index fd7b01d483..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/df6832044b.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "_doc": "Earlier AttendedReportJob run that was still Running when the failing job (c0a1b2c3) tried to start. By the time the diagnostic agent fetches this, the job has Completed normally (EndTime > failing job's StartTime).", - "Data": { - "Key": "b0a1b2c3-d4e5-4678-9012-3456789abcde", - "StartTime": "2026-05-12T10:14:50.000Z", - "EndTime": "2026-05-12T10:15:50.412Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "TimeTrigger", - "BatchExecutionKey": "8c1d6b30-09a2-4b5c-8e7f-1a2b3c4d5e6a", - "Info": "", - "CreationTime": "2026-05-12T10:14:49.852Z", - "ReleaseName": "AttendedReportJob", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 4670012, - "OrganizationUnitId": 7791617, - "FolderKey": "a1b2c3d4-e5f6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-12T10:15:50.412Z", - "JobError": null, - "ErrorCode": null, - "TraceId": "b0a1b2c3-d4e5-4678-9012-3456789abcde", - "Id": 670910198 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/e9b4a8abc2.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/e9b4a8abc2.json deleted file mode 100644 index 41c940aa7e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/e9b4a8abc2.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c0a1b2c3-d4e5-4678-9012-3456789abcde", - "State": "Faulted", - "ProcessName": "AttendedReportJob", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-12T10:15:00.500Z", - "EndTime": "2026-05-12T10:15:01.218Z", - "CreationTime": "2026-05-12T10:15:00.302Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/f5d1ebe209.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/f5d1ebe209.json deleted file mode 100644 index a041583048..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/f5d1ebe209.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-e5f6-7890-1234-56789abcdef0", - "Name": "ForegroundDemo", - "Path": "ForegroundDemo", - "Description": "Demo folder hosting the AttendedReportJob process (foreground / UI-interactive).", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/manifest.json deleted file mode 100644 index d7d3389ad0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-foreground-already-running scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "f5d1ebe209.json" - }, - { - "match": "or jobs list --folder-key \"a1b2c3d4-e5f6-7890-1234-56789abcdef0\" --state Faulted --output json", - "file": "e9b4a8abc2.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-e5f6-7890-1234-56789abcdef0 --state Faulted --output json", - "file": "e9b4a8abc2.json" - }, - { - "match": "or jobs list --folder-key \"a1b2c3d4-e5f6-7890-1234-56789abcdef0\" --output json", - "file": "3603e33798.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-e5f6-7890-1234-56789abcdef0 --output json", - "file": "3603e33798.json" - }, - { - "match": "or jobs get \"c0a1b2c3-d4e5-4678-9012-3456789abcde\" --output json", - "file": "c8f5871c1b.json" - }, - { - "match": "or jobs get c0a1b2c3-d4e5-4678-9012-3456789abcde --output json", - "file": "c8f5871c1b.json" - }, - { - "match": "or jobs get \"b0a1b2c3-d4e5-4678-9012-3456789abcde\" --output json", - "file": "df6832044b.json" - }, - { - "match": "or jobs get b0a1b2c3-d4e5-4678-9012-3456789abcde --output json", - "file": "df6832044b.json" - }, - { - "match": "or jobs logs \"c0a1b2c3-d4e5-4678-9012-3456789abcde\" --level Error --output json", - "file": "1e29694194.json" - }, - { - "match": "or jobs logs c0a1b2c3-d4e5-4678-9012-3456789abcde --level Error --output json", - "file": "1e29694194.json" - }, - { - "match": "or jobs logs \"c0a1b2c3-d4e5-4678-9012-3456789abcde\" --output json", - "file": "dd51f4fc27.json" - }, - { - "match": "or jobs logs c0a1b2c3-d4e5-4678-9012-3456789abcde --output json", - "file": "dd51f4fc27.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/uip-fixture.json new file mode 100644 index 0000000000..fcab416a48 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/data/uip-fixture.json @@ -0,0 +1,105 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-e5f6-7890-1234-56789abcdef0", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c0a1b2c3-d4e5-4678-9012-3456789abcde\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"AttendedReportJob\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-12T10:15:00.500Z\",\r\n \"EndTime\": \"2026-05-12T10:15:01.218Z\",\r\n \"CreationTime\": \"2026-05-12T10:15:00.302Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c0a1b2c3-d4e5-4678-9012-3456789abcde", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T10:15:01.215Z\",\r\n \"Message\": \"Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\\r\\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T10:15:01.190Z\",\r\n \"Message\": \"[Robot] Cannot start foreground job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' (ReleaseName=AttendedReportJob). Another foreground job ('b0a1b2c3-d4e5-4678-9012-3456789abcde', ReleaseName=AttendedReportJob, started 2026-05-12T10:14:50.000Z) is currently running on this session. Only one foreground process can run at a time.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-e5f6-7890-1234-56789abcdef0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"c0a1b2c3-d4e5-4678-9012-3456789abcde\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"AttendedReportJob\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T10:15:00.500Z\",\n \"EndTime\": \"2026-05-12T10:15:01.218Z\",\n \"CreationTime\": \"2026-05-12T10:15:00.302Z\"\n },\n {\n \"Key\": \"b0a1b2c3-d4e5-4678-9012-3456789abcde\",\n \"State\": \"Successful\",\n \"ProcessName\": \"AttendedReportJob\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-12T10:14:50.000Z\",\n \"EndTime\": \"2026-05-12T10:15:50.412Z\",\n \"CreationTime\": \"2026-05-12T10:14:49.852Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c0a1b2c3-d4e5-4678-9012-3456789abcde", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c0a1b2c3-d4e5-4678-9012-3456789abcde\",\r\n \"StartTime\": \"2026-05-12T10:15:00.500Z\",\r\n \"EndTime\": \"2026-05-12T10:15:01.218Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"9d2e7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f\",\r\n \"Info\": \"System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\\r\\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-05-12T10:15:00.302Z\",\r\n \"ReleaseName\": \"AttendedReportJob\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 4670012,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"a1b2c3d4-e5f6-7890-1234-56789abcdef0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-12T10:15:01.218Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c0a1b2c3-d4e5-4678-9012-3456789abcde\",\r\n \"Id\": 670910207\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b0a1b2c3-d4e5-4678-9012-3456789abcde", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobGet\",\n \"Data\": {\n \"Key\": \"b0a1b2c3-d4e5-4678-9012-3456789abcde\",\n \"StartTime\": \"2026-05-12T10:14:50.000Z\",\n \"EndTime\": \"2026-05-12T10:15:50.412Z\",\n \"State\": \"Successful\",\n \"JobPriority\": \"Normal\",\n \"SpecificPriorityValue\": 50,\n \"Source\": \"Schedule\",\n \"SourceType\": \"TimeTrigger\",\n \"BatchExecutionKey\": \"8c1d6b30-09a2-4b5c-8e7f-1a2b3c4d5e6a\",\n \"Info\": \"\",\n \"CreationTime\": \"2026-05-12T10:14:49.852Z\",\n \"ReleaseName\": \"AttendedReportJob\",\n \"Type\": \"Unattended\",\n \"EnvironmentVariables\": \"\",\n \"OutputArguments\": \"{}\",\n \"HostMachineName\": \"MOCK-HOST\",\n \"HasMediaRecorded\": false,\n \"HasVideoRecorded\": false,\n \"RuntimeType\": \"Unattended\",\n \"RequiresUserInteraction\": true,\n \"ReleaseVersionId\": 4670012,\n \"OrganizationUnitId\": 7791617,\n \"FolderKey\": \"a1b2c3d4-e5f6-7890-1234-56789abcdef0\",\n \"Reference\": \"\",\n \"ProcessType\": \"Process\",\n \"ResumeOnSameContext\": false,\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\n \"OrchestratorUserIdentity\": \"user1\",\n \"RemoteControlAccess\": \"None\",\n \"LastModificationTime\": \"2026-05-12T10:15:50.412Z\",\n \"JobError\": null,\n \"ErrorCode\": null,\n \"TraceId\": \"b0a1b2c3-d4e5-4678-9012-3456789abcde\",\n \"Id\": 670910198\n }\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c0a1b2c3-d4e5-4678-9012-3456789abcde", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T10:15:01.215Z\",\r\n \"Message\": \"Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\\r\\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-12T10:15:01.190Z\",\r\n \"Message\": \"[Robot] Cannot start foreground job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' (ReleaseName=AttendedReportJob). Another foreground job ('b0a1b2c3-d4e5-4678-9012-3456789abcde', ReleaseName=AttendedReportJob, started 2026-05-12T10:14:50.000Z) is currently running on this session. Only one foreground process can run at a time.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T10:15:00.510Z\",\r\n \"Message\": \"Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' (ReleaseName=AttendedReportJob) entering Running state on MOCK-HOST.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-12T10:15:00.310Z\",\r\n \"Message\": \"Job 'c0a1b2c3-d4e5-4678-9012-3456789abcde' created. Source=Manual, ReleaseName=AttendedReportJob, RequiresUserInteraction=True.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-7890-1234-56789abcdef0\",\r\n \"Name\": \"ForegroundDemo\",\r\n \"Path\": \"ForegroundDemo\",\r\n \"Description\": \"Demo folder hosting the AttendedReportJob process (foreground / UI-interactive).\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/task.yaml index 981d2fc986..aa01be35cf 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-already-running/task.yaml @@ -23,22 +23,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/README.md new file mode 100644 index 0000000000..05c6c38444 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/README.md @@ -0,0 +1,82 @@ +# Rpa Foreground Misconfigured — Non-UI Workflow Wrongly Marked Foreground + +This scenario reproduces a runtime fault caused by **a workflow that does +no UI interaction being incorrectly published as a foreground process**. +When another foreground job runs against the same Robot session, the +Robot rejects the misconfigured one with: + +``` +System.InvalidOperationException: A foreground process is already running. +Only one foreground process can run at a time. +``` + +The exception is the **same family** as the sibling +`rpa-foreground-already-running` scenario — the cause-branch and +correct fix are different. + +## What this scenario uncovers + +**Root Cause:** The `ExpenseValidation` workflow's `Main.xaml` +contains only `LogMessage` and `Delay` activities — **no UI +automation activities at all**. Yet `project.json` declares +`runtimeOptions.requiresUserInteraction: true`, so the Robot treats it +as a foreground process. It collides with a legitimately foreground +`AttendedReportJob` job that was running on the same Robot session and +faults at start. + +This maps to: +`references/products/orchestrator/playbooks/foreground-already-running.md` +(the **"Process does not actually need UI interaction"** fix branch). + +## How this differs from `rpa-foreground-already-running` + +Both scenarios fault with the same `System.InvalidOperationException`. +The agent must inspect the **source code** to disambiguate the correct +fix: + +| Dimension | `rpa-foreground-already-running` | `rpa-foreground-misconfigured` (this) | +|---|---|---| +| Failing process | `AttendedReportJob` (legitimately foreground) | `ExpenseValidation` (workflow does no UI work) | +| Blocking job | another `AttendedReportJob` (scheduled, overlap) | `AttendedReportJob` (active, holding the slot) | +| Cause-branch | Two foreground triggers overlap | One workflow is misconfigured as foreground | +| Primary fix | Stagger triggers / "Run only one job at a time" | **Set `requiresUserInteraction: false` in `project.json` / Studio "Starts in Background: Yes"** | +| Required agent action | Read job records to find overlap | **Read `Main.xaml` to confirm no UI activities, then read `project.json` to find the misconfig** | +| Wrong answer | Suggesting the misconfig fix | Suggesting only trigger sequencing without flagging the misconfig | + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | snapshot of `skills/uipath-troubleshoot/fixtures/foreground-already-running/ExpenseValidation/` — the misconfigured project | + +> **Note on fixtures.** Like sibling synthetic scenarios, the fixtures +> were authored from the documented playbook rather than captured from +> a real `.investigation/` session. Regenerate via +> `_shared/scripts/generate_scenario.py` from a real failed-job +> session before treating this test's score as a regression signal. + +## Success criteria + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `orchestrator/foreground-already-running` (not the + sibling maestro `#1230` playbook) +- Agent inspected `Main.xaml` and identified no UI activities +- Agent inspected `project.json` and identified + `requiresUserInteraction: true` as the misconfiguration +- Conclusion recommends the `project.json` flip / Studio "Starts in + Background: Yes" as the **primary** fix, targeted at + `ExpenseValidation` (not at `AttendedReportJob`) + +## Regenerating from a real session + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation \ + --project \ + --transcript \ + --scenario-name rpa-foreground-misconfigured --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/0966b64df7.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/0966b64df7.json deleted file mode 100644 index 556866e775..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/0966b64df7.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d1e2f3a4-b5c6-4789-abcd-ef0123456789", - "StartTime": "2026-05-13T14:20:45.500Z", - "EndTime": "2026-05-13T14:20:46.218Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7e1c6b30-19a2-4b5c-8e7f-1a2b3c4d5e6f", - "Info": "System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\r\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-13T14:20:45.302Z", - "ReleaseName": "ExpenseValidation", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 4670220, - "OrganizationUnitId": 7791617, - "FolderKey": "b2c3d4e5-f6a7-8901-2345-6789abcdef01", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T14:20:46.218Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d1e2f3a4-b5c6-4789-abcd-ef0123456789", - "Id": 670910311 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/5123bf7491.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/5123bf7491.json deleted file mode 100644 index d8501988d0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/5123bf7491.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T14:20:46.215Z", - "Message": "Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\r\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T14:20:46.190Z", - "Message": "[Robot] Cannot start foreground job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' (ReleaseName=ExpenseValidation). Another foreground job ('e1f2a3b4-c5d6-4789-abcd-ef0123456789', ReleaseName=AttendedReportJob, started 2026-05-13T14:20:30.000Z) is currently running on this session. Only one foreground process can run at a time." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T14:20:45.510Z", - "Message": "Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' (ReleaseName=ExpenseValidation) entering Running state on MOCK-HOST." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T14:20:45.310Z", - "Message": "Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' created. Source=Manual, ReleaseName=ExpenseValidation, RequiresUserInteraction=True." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/7d856481cf.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/7d856481cf.json deleted file mode 100644 index 8401770b6b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/7d856481cf.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "d1e2f3a4-b5c6-4789-abcd-ef0123456789", - "State": "Faulted", - "ProcessName": "ExpenseValidation", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T14:20:45.500Z", - "EndTime": "2026-05-13T14:20:46.218Z", - "CreationTime": "2026-05-13T14:20:45.302Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/9babdd7530.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/9babdd7530.json deleted file mode 100644 index 4d8bd526ca..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/9babdd7530.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "b2c3d4e5-f6a7-8901-2345-6789abcdef01", - "Name": "ComputeJobs", - "Path": "ComputeJobs", - "Description": "Folder hosting compute-heavy automations including the (misconfigured) ExpenseValidation process and the foreground AttendedReportJob process.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/README.md deleted file mode 100644 index d92d37c753..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/README.md +++ /dev/null @@ -1,81 +0,0 @@ -# Rpa Foreground Misconfigured — Non-UI Workflow Wrongly Marked Foreground - -This scenario reproduces a runtime fault caused by **a workflow that does -no UI interaction being incorrectly published as a foreground process**. -When another foreground job runs against the same Robot session, the -Robot rejects the misconfigured one with: - -``` -System.InvalidOperationException: A foreground process is already running. -Only one foreground process can run at a time. -``` - -The exception is the **same family** as the sibling -`rpa-foreground-already-running` scenario — the cause-branch and -correct fix are different. - -## What this scenario uncovers - -**Root Cause:** The `ExpenseValidation` workflow's `Main.xaml` -contains only `LogMessage` and `Delay` activities — **no UI -automation activities at all**. Yet `project.json` declares -`runtimeOptions.requiresUserInteraction: true`, so the Robot treats it -as a foreground process. It collides with a legitimately foreground -`AttendedReportJob` job that was running on the same Robot session and -faults at start. - -This maps to: -`references/products/orchestrator/playbooks/foreground-already-running.md` -(the **"Process does not actually need UI interaction"** fix branch). - -## How this differs from `rpa-foreground-already-running` - -Both scenarios fault with the same `System.InvalidOperationException`. -The agent must inspect the **source code** to disambiguate the correct -fix: - -| Dimension | `rpa-foreground-already-running` | `rpa-foreground-misconfigured` (this) | -|---|---|---| -| Failing process | `AttendedReportJob` (legitimately foreground) | `ExpenseValidation` (workflow does no UI work) | -| Blocking job | another `AttendedReportJob` (scheduled, overlap) | `AttendedReportJob` (active, holding the slot) | -| Cause-branch | Two foreground triggers overlap | One workflow is misconfigured as foreground | -| Primary fix | Stagger triggers / "Run only one job at a time" | **Set `requiresUserInteraction: false` in `project.json` / Studio "Starts in Background: Yes"** | -| Required agent action | Read job records to find overlap | **Read `Main.xaml` to confirm no UI activities, then read `project.json` to find the misconfig** | -| Wrong answer | Suggesting the misconfig fix | Suggesting only trigger sequencing without flagging the misconfig | - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | snapshot of `skills/uipath-troubleshoot/fixtures/foreground-already-running/ExpenseValidation/` — the misconfigured project | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -> **Note on fixtures.** Like sibling synthetic scenarios, the fixtures -> were authored from the documented playbook rather than captured from -> a real `.investigation/` session. Regenerate via -> `_shared/scripts/generate_scenario.py` from a real failed-job -> session before treating this test's score as a regression signal. - -## Success criteria - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `orchestrator/foreground-already-running` (not the - sibling maestro `#1230` playbook) -- Agent inspected `Main.xaml` and identified no UI activities -- Agent inspected `project.json` and identified - `requiresUserInteraction: true` as the misconfiguration -- Conclusion recommends the `project.json` flip / Studio "Starts in - Background: Yes" as the **primary** fix, targeted at - `ExpenseValidation` (not at `AttendedReportJob`) - -## Regenerating from a real session - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation \ - --project \ - --transcript \ - --scenario-name rpa-foreground-misconfigured --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/c95ae424b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/c95ae424b6.json deleted file mode 100644 index 31b7c255d6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/c95ae424b6.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "_doc": "Legitimately-foreground AttendedReportJob job that was holding the slot when ExpenseValidation tried to start. By the time the agent fetches this, the job has completed normally.", - "Data": { - "Key": "e1f2a3b4-c5d6-4789-abcd-ef0123456789", - "StartTime": "2026-05-13T14:20:30.000Z", - "EndTime": "2026-05-13T14:21:30.412Z", - "State": "Successful", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "TimeTrigger", - "BatchExecutionKey": "6d2c5a20-08b1-4a4d-8e6e-0a1b2c3d4e5f", - "Info": "", - "CreationTime": "2026-05-13T14:20:29.851Z", - "ReleaseName": "AttendedReportJob", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": true, - "ReleaseVersionId": 4670012, - "OrganizationUnitId": 7791617, - "FolderKey": "b2c3d4e5-f6a7-8901-2345-6789abcdef01", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\USER1", - "OrchestratorUserIdentity": "user1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T14:21:30.412Z", - "JobError": null, - "ErrorCode": null, - "TraceId": "e1f2a3b4-c5d6-4789-abcd-ef0123456789", - "Id": 670910298 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/d5798d9cc0.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/d5798d9cc0.json deleted file mode 100644 index b51172a0ae..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/d5798d9cc0.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent jobs in ComputeJobs. The failing ExpenseValidation (d1e2f3a4) tried to start while AttendedReportJob (e1f2a3b4) was Running. Notice both have foreground intent but only AttendedReportJob is legitimate per its source code.", - "Data": [ - { - "Key": "d1e2f3a4-b5c6-4789-abcd-ef0123456789", - "State": "Faulted", - "ProcessName": "ExpenseValidation", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T14:20:45.500Z", - "EndTime": "2026-05-13T14:20:46.218Z", - "CreationTime": "2026-05-13T14:20:45.302Z" - }, - { - "Key": "e1f2a3b4-c5d6-4789-abcd-ef0123456789", - "State": "Successful", - "ProcessName": "AttendedReportJob", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-13T14:20:30.000Z", - "EndTime": "2026-05-13T14:21:30.412Z", - "CreationTime": "2026-05-13T14:20:29.851Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/f6ca393b74.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/f6ca393b74.json deleted file mode 100644 index d19f0705a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/f6ca393b74.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T14:20:46.215Z", - "Message": "Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\r\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T14:20:46.190Z", - "Message": "[Robot] Cannot start foreground job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' (ReleaseName=ExpenseValidation). Another foreground job ('e1f2a3b4-c5d6-4789-abcd-ef0123456789', ReleaseName=AttendedReportJob, started 2026-05-13T14:20:30.000Z) is currently running on this session. Only one foreground process can run at a time." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/manifest.json deleted file mode 100644 index 41376c793e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/m/r/manifest.json +++ /dev/null @@ -1,68 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-foreground-misconfigured scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "9babdd7530.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-8901-2345-6789abcdef01\" --state Faulted --output json", - "file": "7d856481cf.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-8901-2345-6789abcdef01 --state Faulted --output json", - "file": "7d856481cf.json" - }, - { - "match": "or jobs list --folder-key \"b2c3d4e5-f6a7-8901-2345-6789abcdef01\" --output json", - "file": "d5798d9cc0.json" - }, - { - "match": "or jobs list --folder-key b2c3d4e5-f6a7-8901-2345-6789abcdef01 --output json", - "file": "d5798d9cc0.json" - }, - { - "match": "or jobs get \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\" --output json", - "file": "0966b64df7.json" - }, - { - "match": "or jobs get d1e2f3a4-b5c6-4789-abcd-ef0123456789 --output json", - "file": "0966b64df7.json" - }, - { - "match": "or jobs get \"e1f2a3b4-c5d6-4789-abcd-ef0123456789\" --output json", - "file": "c95ae424b6.json" - }, - { - "match": "or jobs get e1f2a3b4-c5d6-4789-abcd-ef0123456789 --output json", - "file": "c95ae424b6.json" - }, - { - "match": "or jobs logs \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\" --level Error --output json", - "file": "f6ca393b74.json" - }, - { - "match": "or jobs logs d1e2f3a4-b5c6-4789-abcd-ef0123456789 --level Error --output json", - "file": "f6ca393b74.json" - }, - { - "match": "or jobs logs \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\" --output json", - "file": "5123bf7491.json" - }, - { - "match": "or jobs logs d1e2f3a4-b5c6-4789-abcd-ef0123456789 --output json", - "file": "5123bf7491.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: unmatched uip commands return an empty array." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/uip-fixture.json new file mode 100644 index 0000000000..a5219d6fa9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/data/uip-fixture.json @@ -0,0 +1,105 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-8901-2345-6789abcdef01", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ExpenseValidation\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Manual\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"MachineKey\": \"\",\r\n \"StartTime\": \"2026-05-13T14:20:45.500Z\",\r\n \"EndTime\": \"2026-05-13T14:20:46.218Z\",\r\n \"CreationTime\": \"2026-05-13T14:20:45.302Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d1e2f3a4-b5c6-4789-abcd-ef0123456789", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T14:20:46.215Z\",\r\n \"Message\": \"Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\\r\\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T14:20:46.190Z\",\r\n \"Message\": \"[Robot] Cannot start foreground job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' (ReleaseName=ExpenseValidation). Another foreground job ('e1f2a3b4-c5d6-4789-abcd-ef0123456789', ReleaseName=AttendedReportJob, started 2026-05-13T14:20:30.000Z) is currently running on this session. Only one foreground process can run at a time.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "b2c3d4e5-f6a7-8901-2345-6789abcdef01", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"ExpenseValidation\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-13T14:20:45.500Z\",\n \"EndTime\": \"2026-05-13T14:20:46.218Z\",\n \"CreationTime\": \"2026-05-13T14:20:45.302Z\"\n },\n {\n \"Key\": \"e1f2a3b4-c5d6-4789-abcd-ef0123456789\",\n \"State\": \"Successful\",\n \"ProcessName\": \"AttendedReportJob\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-13T14:20:30.000Z\",\n \"EndTime\": \"2026-05-13T14:21:30.412Z\",\n \"CreationTime\": \"2026-05-13T14:20:29.851Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d1e2f3a4-b5c6-4789-abcd-ef0123456789", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\",\r\n \"StartTime\": \"2026-05-13T14:20:45.500Z\",\r\n \"EndTime\": \"2026-05-13T14:20:46.218Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7e1c6b30-19a2-4b5c-8e7f-1a2b3c4d5e6f\",\r\n \"Info\": \"System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\\r\\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-05-13T14:20:45.302Z\",\r\n \"ReleaseName\": \"ExpenseValidation\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 4670220,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"b2c3d4e5-f6a7-8901-2345-6789abcdef01\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\r\n \"OrchestratorUserIdentity\": \"user1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T14:20:46.218Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d1e2f3a4-b5c6-4789-abcd-ef0123456789\",\r\n \"Id\": 670910311\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e1f2a3b4-c5d6-4789-abcd-ef0123456789", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobGet\",\n \"Data\": {\n \"Key\": \"e1f2a3b4-c5d6-4789-abcd-ef0123456789\",\n \"StartTime\": \"2026-05-13T14:20:30.000Z\",\n \"EndTime\": \"2026-05-13T14:21:30.412Z\",\n \"State\": \"Successful\",\n \"JobPriority\": \"Normal\",\n \"SpecificPriorityValue\": 50,\n \"Source\": \"Schedule\",\n \"SourceType\": \"TimeTrigger\",\n \"BatchExecutionKey\": \"6d2c5a20-08b1-4a4d-8e6e-0a1b2c3d4e5f\",\n \"Info\": \"\",\n \"CreationTime\": \"2026-05-13T14:20:29.851Z\",\n \"ReleaseName\": \"AttendedReportJob\",\n \"Type\": \"Unattended\",\n \"EnvironmentVariables\": \"\",\n \"OutputArguments\": \"{}\",\n \"HostMachineName\": \"MOCK-HOST\",\n \"HasMediaRecorded\": false,\n \"HasVideoRecorded\": false,\n \"RuntimeType\": \"Unattended\",\n \"RequiresUserInteraction\": true,\n \"ReleaseVersionId\": 4670012,\n \"OrganizationUnitId\": 7791617,\n \"FolderKey\": \"b2c3d4e5-f6a7-8901-2345-6789abcdef01\",\n \"Reference\": \"\",\n \"ProcessType\": \"Process\",\n \"ResumeOnSameContext\": false,\n \"LocalSystemAccount\": \"UIPATH\\\\USER1\",\n \"OrchestratorUserIdentity\": \"user1\",\n \"RemoteControlAccess\": \"None\",\n \"LastModificationTime\": \"2026-05-13T14:21:30.412Z\",\n \"JobError\": null,\n \"ErrorCode\": null,\n \"TraceId\": \"e1f2a3b4-c5d6-4789-abcd-ef0123456789\",\n \"Id\": 670910298\n }\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "d1e2f3a4-b5c6-4789-abcd-ef0123456789", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T14:20:46.215Z\",\r\n \"Message\": \"Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' failed to start. System.InvalidOperationException: A foreground process is already running. Only one foreground process can run at a time.\\r\\n at UiPath.Robot.Service.Hosting.ForegroundJobGuard.EnsureCanStart(JobStartRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T14:20:46.190Z\",\r\n \"Message\": \"[Robot] Cannot start foreground job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' (ReleaseName=ExpenseValidation). Another foreground job ('e1f2a3b4-c5d6-4789-abcd-ef0123456789', ReleaseName=AttendedReportJob, started 2026-05-13T14:20:30.000Z) is currently running on this session. Only one foreground process can run at a time.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T14:20:45.510Z\",\r\n \"Message\": \"Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' (ReleaseName=ExpenseValidation) entering Running state on MOCK-HOST.\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T14:20:45.310Z\",\r\n \"Message\": \"Job 'd1e2f3a4-b5c6-4789-abcd-ef0123456789' created. Source=Manual, ReleaseName=ExpenseValidation, RequiresUserInteraction=True.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"b2c3d4e5-f6a7-8901-2345-6789abcdef01\",\r\n \"Name\": \"ComputeJobs\",\r\n \"Path\": \"ComputeJobs\",\r\n \"Description\": \"Folder hosting compute-heavy automations including the (misconfigured) ExpenseValidation process and the foreground AttendedReportJob process.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/task.yaml index 7591d6edb9..5531d72e8b 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-foreground-misconfigured/task.yaml @@ -28,22 +28,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/README.md new file mode 100644 index 0000000000..ce5abfeb24 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/README.md @@ -0,0 +1,70 @@ +# Rpa Job Killed By Autocancel Trigger — Time-Budget Cancellation + +This scenario reproduces a runtime fault caused by **an auto-cancel +trigger force-killing a running job after its configured time budget +was exceeded**. The job ends with: + +``` +System.Exception: Job stopped with an unexpected exit code: 0x40010004 +``` + +(Windows `DBG_TERMINATE_PROCESS`.) + +## What this scenario uncovers + +**Root Cause:** Job `LongRunningProcess` (key `c3c0b2c3-...`) was +running normally when the `kill-after-10min` auto-cancel trigger +fired at exactly +600 seconds (the configured time budget). The +trigger calls `Stop` with `strategy=Kill` on jobs that exceed their +budget; this run was on the slow end (still healthy, but past the +trigger threshold). The audit log records the actor as `System` — +no human, no service account — with `eventDetails` referencing the +trigger name and time-budget reason. + +This maps to: +`references/products/orchestrator/playbooks/job-stopped-exit-code-0x40010004.md` +(the "Operator Kill from Orchestrator" branch, system-trigger +origin). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal `LongRunningProcess` project — single 5-minute Delay activity | + +The Killing transition timestamp at exactly +600s (matching the +trigger's `timeBudgetSeconds: 600`) is the strongest signal. The +audit event's `actorName: "System"`, `eventDetails.triggerType: +"AutoCancel"`, and `reason: "TimeBudgetExceeded"` confirm the +trigger-initiated kill. + +## How this differs from the sibling Kill variants + +| Scenario | Actor | Surface signal | +|---|---|---| +| `rpa-job-killed-by-operator-ui` | Human user via Orchestrator UI | `actorEmail` is a real user; `clientInfo.ipAddress` present | +| `rpa-job-killed-by-watchdog-account` | Service account via API | `actorName` is a service account; `clientInfo` absent; `origin: API` | +| **this** — `rpa-job-killed-by-autocancel-trigger` | System-initiated cancellation trigger | `actorName` is `System`; `eventDetails.triggerType: AutoCancel`; Killing at exactly +budgetSeconds | + +A correct agent matches the **0x40010004 playbook → Operator Kill +branch** and distinguishes the trigger-initiated cancel from a +human or service-account kill by inspecting `actorName: "System"` +and `eventDetails.triggerType: "AutoCancel"`. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `orchestrator/job-stopped-exit-code-0x40010004` +- Agent ran `uip or jobs history` and observed the + `Running → Killing → Faulted` transition at +600s +- Agent ran `uip admin audit tenant events --search ` and + identified the `actorName: "System"` with trigger details +- Conclusion recommends a fix from the acceptable list (raise time + budget, scope trigger more narrowly, fix the workflow to fit + budget, or remove trigger) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/0daab04c98.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/0daab04c98.json deleted file mode 100644 index cfb84637d1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/0daab04c98.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent jobs in KillScenarios. The failing job (c3c0b2c3) is preceded by several successful LongRunningProcess runs — no overlap, no other Killing state — so the kill is not a recurring pattern.", - "Data": [ - { - "Key": "c3c0b2c3-d4e5-4678-9012-3456789ccccc", - "State": "Faulted", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:10:00.875Z", - "CreationTime": "2026-05-19T08:00:00.302Z" - }, - { - "Key": "9aa0b2c3-d4e5-4678-9012-3456789aaaa1", - "State": "Successful", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T08:00:00.500Z", - "EndTime": "2026-05-18T08:05:00.812Z", - "CreationTime": "2026-05-18T08:00:00.295Z" - }, - { - "Key": "8aa0b2c3-d4e5-4678-9012-3456789aaaa2", - "State": "Successful", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-17T08:00:00.500Z", - "EndTime": "2026-05-17T08:05:00.755Z", - "CreationTime": "2026-05-17T08:00:00.310Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/63044a0ebf.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/63044a0ebf.json deleted file mode 100644 index 7728fc0207..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/63044a0ebf.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Name": "KillScenarios", - "Path": "KillScenarios", - "Description": "Folder hosting LongRunningProcess — long-running job used to exercise the 0x40010004 (Job killed) playbook variants.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/7e2ed776f6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/7e2ed776f6.json deleted file mode 100644 index 88bb00fabb..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/7e2ed776f6.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Faulted-only view: just the failing LongRunningProcess job.", - "Data": [ - { - "Key": "c3c0b2c3-d4e5-4678-9012-3456789ccccc", - "State": "Faulted", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:10:00.875Z", - "CreationTime": "2026-05-19T08:00:00.302Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/94ddcd3e19.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/94ddcd3e19.json deleted file mode 100644 index d9197c4266..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/94ddcd3e19.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "AuditTenantEvents", - "_doc": "Audit query returns the Job.Stop event matching the failing job key. The actor is 'System' (no human/service account identity) — eventDetails references the auto-cancel trigger name and the time budget that was exceeded. clientInfo is absent (system action).", - "Data": { - "auditEvents": [ - { - "id": "a6f1e62-3456-4cde-9def-3c4d5e6f7g8h", - "createdOn": "2026-05-19T08:10:00.105Z", - "organizationId": "11111111-2222-3333-4444-555555555555", - "organizationName": "Acme Corp", - "tenantId": "22222222-3333-4444-5555-666666666666", - "tenantName": "Acme-Prod", - "actorId": "00000000-0000-0000-0000-000000000000", - "actorName": "System", - "actorEmail": null, - "eventType": "Job stopped", - "eventSource": "Orchestrator", - "eventTarget": "Job", - "eventDetails": "{\"jobKey\":\"c3c0b2c3-d4e5-4678-9012-3456789ccccc\",\"processName\":\"LongRunningProcess\",\"strategy\":\"Kill\",\"origin\":\"Trigger\",\"folderKey\":\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\"folderName\":\"KillScenarios\",\"actorType\":\"System\",\"triggerName\":\"kill-after-10min\",\"triggerType\":\"AutoCancel\",\"timeBudgetSeconds\":600,\"reason\":\"TimeBudgetExceeded\"}", - "eventSummary": "Job 'LongRunningProcess' (c3c0b2c3-d4e5-4678-9012-3456789ccccc) was stopped (strategy Kill) by auto-cancel trigger 'kill-after-10min' (time budget 600s exceeded)", - "status": 0 - } - ], - "next": null, - "previous": null - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/README.md deleted file mode 100644 index 991b8652f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/README.md +++ /dev/null @@ -1,69 +0,0 @@ -# Rpa Job Killed By Autocancel Trigger — Time-Budget Cancellation - -This scenario reproduces a runtime fault caused by **an auto-cancel -trigger force-killing a running job after its configured time budget -was exceeded**. The job ends with: - -``` -System.Exception: Job stopped with an unexpected exit code: 0x40010004 -``` - -(Windows `DBG_TERMINATE_PROCESS`.) - -## What this scenario uncovers - -**Root Cause:** Job `LongRunningProcess` (key `c3c0b2c3-...`) was -running normally when the `kill-after-10min` auto-cancel trigger -fired at exactly +600 seconds (the configured time budget). The -trigger calls `Stop` with `strategy=Kill` on jobs that exceed their -budget; this run was on the slow end (still healthy, but past the -trigger threshold). The audit log records the actor as `System` — -no human, no service account — with `eventDetails` referencing the -trigger name and time-budget reason. - -This maps to: -`references/products/orchestrator/playbooks/job-stopped-exit-code-0x40010004.md` -(the "Operator Kill from Orchestrator" branch, system-trigger -origin). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | minimal `LongRunningProcess` project — single 5-minute Delay activity | -| `fixtures/mocks/responses/*.json` | synthetic `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The Killing transition timestamp at exactly +600s (matching the -trigger's `timeBudgetSeconds: 600`) is the strongest signal. The -audit event's `actorName: "System"`, `eventDetails.triggerType: -"AutoCancel"`, and `reason: "TimeBudgetExceeded"` confirm the -trigger-initiated kill. - -## How this differs from the sibling Kill variants - -| Scenario | Actor | Surface signal | -|---|---|---| -| `rpa-job-killed-by-operator-ui` | Human user via Orchestrator UI | `actorEmail` is a real user; `clientInfo.ipAddress` present | -| `rpa-job-killed-by-watchdog-account` | Service account via API | `actorName` is a service account; `clientInfo` absent; `origin: API` | -| **this** — `rpa-job-killed-by-autocancel-trigger` | System-initiated cancellation trigger | `actorName` is `System`; `eventDetails.triggerType: AutoCancel`; Killing at exactly +budgetSeconds | - -A correct agent matches the **0x40010004 playbook → Operator Kill -branch** and distinguishes the trigger-initiated cancel from a -human or service-account kill by inspecting `actorName: "System"` -and `eventDetails.triggerType: "AutoCancel"`. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `orchestrator/job-stopped-exit-code-0x40010004` -- Agent ran `uip or jobs history` and observed the - `Running → Killing → Faulted` transition at +600s -- Agent ran `uip admin audit tenant events --search ` and - identified the `actorName: "System"` with trigger details -- Conclusion recommends a fix from the acceptable list (raise time - budget, scope trigger more narrowly, fix the workflow to fit - budget, or remove trigger) diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/a3af837b80.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/a3af837b80.json deleted file mode 100644 index 97b5a9d276..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/a3af837b80.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "_doc": "State transitions for the failing job. Running for exactly 600s before transitioning to Killing — matches the configured auto-cancel trigger time budget. The Source field on the Killing transition is 'Trigger:kill-after-10min', identifying the system-initiated cancel.", - "Data": [ - { - "State": "Pending", - "Timestamp": "2026-05-19T08:00:00.302Z", - "Reason": "JobCreated" - }, - { - "State": "Running", - "Timestamp": "2026-05-19T08:00:00.500Z", - "Reason": "ExecutorStarted" - }, - { - "State": "Killing", - "Timestamp": "2026-05-19T08:10:00.000Z", - "Reason": "TimeBudgetExceeded", - "Strategy": "Kill", - "Source": "Trigger:kill-after-10min" - }, - { - "State": "Faulted", - "Timestamp": "2026-05-19T08:10:00.875Z", - "Reason": "ExecutorExited", - "ExitCode": "0x40010004" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/be1ccc7600.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/be1ccc7600.json deleted file mode 100644 index fcf2f805db..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/be1ccc7600.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "AuditTenantSources", - "_doc": "Minimal sources discovery payload. Real responses include many more sources; this stub lists only the ones relevant to job-stop investigations so the agent has plausible GUIDs if it filters by --source.", - "Data": [ - { - "id": "33333333-aaaa-4444-bbbb-555555555555", - "name": "Orchestrator", - "targets": [ - { - "id": "44444444-cccc-5555-dddd-666666666666", - "name": "Job" - } - ], - "types": [ - { - "id": "55555555-eeee-6666-ffff-777777777777", - "name": "Job stopped" - } - ] - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/be3c4e785f.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/be3c4e785f.json deleted file mode 100644 index bb46a79ee7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/be3c4e785f.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "_doc": "Execution logs end abruptly mid-Delay after exactly 10 minutes — no Job Completed line. Robot Service logs the auto-cancel trigger fire and the abrupt exit. The kill source is recorded as the trigger name.", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-19T08:10:00.875Z", - "Message": "Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed." - }, - { - "Level": "Warn", - "TimeStamp": "2026-05-19T08:10:00.020Z", - "Message": "[Robot] Stop request received for job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (strategy=Kill, source=Trigger:kill-after-10min, reason=TimeBudgetExceeded). Terminating executor process." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.520Z", - "Message": "[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.510Z", - "Message": "Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.310Z", - "Message": "Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' created. Source=Schedule, ReleaseName=LongRunningProcess, RequiresUserInteraction=False." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/d2c0fce688.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/d2c0fce688.json deleted file mode 100644 index 7f21da65fb..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/d2c0fce688.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3c0b2c3-d4e5-4678-9012-3456789ccccc", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:10:00.875Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "9f4f7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f", - "Info": "System.Exception: Job stopped with an unexpected exit code: 0x40010004\r\n at UiPath.Robot.Service.Hosting.ExecutorSupervisor.HandleExitAsync(Int32 exitCode, CancellationToken cancellationToken)\r\n at UiPath.Robot.Service.JobManager.FinalizeJobAsync(JobRecord record, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-19T08:00:00.302Z", - "ReleaseName": "LongRunningProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5670099, - "OrganizationUnitId": 7791617, - "FolderKey": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:10:00.875Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3c0b2c3-d4e5-4678-9012-3456789ccccc", - "Id": 670911077 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/manifest.json deleted file mode 100644 index bf4e81760c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/m/r/manifest.json +++ /dev/null @@ -1,93 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-job-killed-by-autocancel-trigger scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "63044a0ebf.json" - }, - { - "match": "or folders list --output json", - "file": "63044a0ebf.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --state Faulted --output json", - "file": "7e2ed776f6.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --state Faulted --output json", - "file": "7e2ed776f6.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --output json", - "file": "0daab04c98.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --output json", - "file": "0daab04c98.json" - }, - { - "match": "or jobs get \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\" --output json", - "file": "d2c0fce688.json" - }, - { - "match": "or jobs get c3c0b2c3-d4e5-4678-9012-3456789ccccc --output json", - "file": "d2c0fce688.json" - }, - { - "match": "or jobs history \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\" --output json", - "file": "a3af837b80.json" - }, - { - "match": "or jobs history c3c0b2c3-d4e5-4678-9012-3456789ccccc --output json", - "file": "a3af837b80.json" - }, - { - "match": "or jobs logs \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\" --level Error --output json", - "file": "be3c4e785f.json" - }, - { - "match": "or jobs logs c3c0b2c3-d4e5-4678-9012-3456789ccccc --level Error --output json", - "file": "be3c4e785f.json" - }, - { - "match": "or jobs logs \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\" --output json", - "file": "be3c4e785f.json" - }, - { - "match": "or jobs logs c3c0b2c3-d4e5-4678-9012-3456789ccccc --output json", - "file": "be3c4e785f.json" - }, - { - "match": "admin audit tenant events", - "file": "94ddcd3e19.json" - }, - { - "match": "admin audit org events", - "file": "94ddcd3e19.json" - }, - { - "match": "admin audit tenant sources", - "file": "be1ccc7600.json" - }, - { - "match": "admin audit org sources", - "file": "be1ccc7600.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { "pattern": "or jobs get c3c0b2c3", "min": 1, "description": "Agent must inspect the failing job to confirm the exit code" }, - { "pattern": "or jobs history c3c0b2c3", "min": 1, "description": "Agent must inspect state history to distinguish operator-initiated Stop from host-side termination" }, - { "pattern": "admin audit", "min": 1, "description": "Agent must query the audit log to identify the actor" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/uip-fixture.json new file mode 100644 index 0000000000..835719e6c5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/data/uip-fixture.json @@ -0,0 +1,166 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\n \"EndTime\": \"2026-05-19T08:10:00.875Z\",\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3c0b2c3-d4e5-4678-9012-3456789ccccc", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobLogs\",\n \"Data\": [\n {\n \"Level\": \"Error\",\n \"TimeStamp\": \"2026-05-19T08:10:00.875Z\",\n \"Message\": \"Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed.\"\n },\n {\n \"Level\": \"Warn\",\n \"TimeStamp\": \"2026-05-19T08:10:00.020Z\",\n \"Message\": \"[Robot] Stop request received for job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (strategy=Kill, source=Trigger:kill-after-10min, reason=TimeBudgetExceeded). Terminating executor process.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.520Z\",\n \"Message\": \"[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.510Z\",\n \"Message\": \"Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.310Z\",\n \"Message\": \"Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' created. Source=Schedule, ReleaseName=LongRunningProcess, RequiresUserInteraction=False.\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\n \"EndTime\": \"2026-05-19T08:10:00.875Z\",\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\"\n },\n {\n \"Key\": \"9aa0b2c3-d4e5-4678-9012-3456789aaaa1\",\n \"State\": \"Successful\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-18T08:00:00.500Z\",\n \"EndTime\": \"2026-05-18T08:05:00.812Z\",\n \"CreationTime\": \"2026-05-18T08:00:00.295Z\"\n },\n {\n \"Key\": \"8aa0b2c3-d4e5-4678-9012-3456789aaaa2\",\n \"State\": \"Successful\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-17T08:00:00.500Z\",\n \"EndTime\": \"2026-05-17T08:05:00.755Z\",\n \"CreationTime\": \"2026-05-17T08:00:00.310Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3c0b2c3-d4e5-4678-9012-3456789ccccc", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\",\r\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T08:10:00.875Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"9f4f7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f\",\r\n \"Info\": \"System.Exception: Job stopped with an unexpected exit code: 0x40010004\\r\\n at UiPath.Robot.Service.Hosting.ExecutorSupervisor.HandleExitAsync(Int32 exitCode, CancellationToken cancellationToken)\\r\\n at UiPath.Robot.Service.JobManager.FinalizeJobAsync(JobRecord record, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\",\r\n \"ReleaseName\": \"LongRunningProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5670099,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:10:00.875Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3c0b2c3-d4e5-4678-9012-3456789ccccc\",\r\n \"Id\": 670911077\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "c3c0b2c3-d4e5-4678-9012-3456789ccccc", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobHistory\",\n \"Data\": [\n {\n \"State\": \"Pending\",\n \"Timestamp\": \"2026-05-19T08:00:00.302Z\",\n \"Reason\": \"JobCreated\"\n },\n {\n \"State\": \"Running\",\n \"Timestamp\": \"2026-05-19T08:00:00.500Z\",\n \"Reason\": \"ExecutorStarted\"\n },\n {\n \"State\": \"Killing\",\n \"Timestamp\": \"2026-05-19T08:10:00.000Z\",\n \"Reason\": \"TimeBudgetExceeded\",\n \"Strategy\": \"Kill\",\n \"Source\": \"Trigger:kill-after-10min\"\n },\n {\n \"State\": \"Faulted\",\n \"Timestamp\": \"2026-05-19T08:10:00.875Z\",\n \"Reason\": \"ExecutorExited\",\n \"ExitCode\": \"0x40010004\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3c0b2c3-d4e5-4678-9012-3456789ccccc", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobLogs\",\n \"Data\": [\n {\n \"Level\": \"Error\",\n \"TimeStamp\": \"2026-05-19T08:10:00.875Z\",\n \"Message\": \"Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed.\"\n },\n {\n \"Level\": \"Warn\",\n \"TimeStamp\": \"2026-05-19T08:10:00.020Z\",\n \"Message\": \"[Robot] Stop request received for job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (strategy=Kill, source=Trigger:kill-after-10min, reason=TimeBudgetExceeded). Terminating executor process.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.520Z\",\n \"Message\": \"[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.510Z\",\n \"Message\": \"Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.310Z\",\n \"Message\": \"Job 'c3c0b2c3-d4e5-4678-9012-3456789ccccc' created. Source=Schedule, ReleaseName=LongRunningProcess, RequiresUserInteraction=False.\"\n }\n ]\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "tenant", + "events" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantEvents\",\n \"Data\": {\n \"auditEvents\": [\n {\n \"id\": \"a6f1e62-3456-4cde-9def-3c4d5e6f7g8h\",\n \"createdOn\": \"2026-05-19T08:10:00.105Z\",\n \"organizationId\": \"11111111-2222-3333-4444-555555555555\",\n \"organizationName\": \"Acme Corp\",\n \"tenantId\": \"22222222-3333-4444-5555-666666666666\",\n \"tenantName\": \"Acme-Prod\",\n \"actorId\": \"00000000-0000-0000-0000-000000000000\",\n \"actorName\": \"System\",\n \"actorEmail\": null,\n \"eventType\": \"Job stopped\",\n \"eventSource\": \"Orchestrator\",\n \"eventTarget\": \"Job\",\n \"eventDetails\": \"{\\\"jobKey\\\":\\\"c3c0b2c3-d4e5-4678-9012-3456789ccccc\\\",\\\"processName\\\":\\\"LongRunningProcess\\\",\\\"strategy\\\":\\\"Kill\\\",\\\"origin\\\":\\\"Trigger\\\",\\\"folderKey\\\":\\\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\\\",\\\"folderName\\\":\\\"KillScenarios\\\",\\\"actorType\\\":\\\"System\\\",\\\"triggerName\\\":\\\"kill-after-10min\\\",\\\"triggerType\\\":\\\"AutoCancel\\\",\\\"timeBudgetSeconds\\\":600,\\\"reason\\\":\\\"TimeBudgetExceeded\\\"}\",\n \"eventSummary\": \"Job 'LongRunningProcess' (c3c0b2c3-d4e5-4678-9012-3456789ccccc) was stopped (strategy Kill) by auto-cancel trigger 'kill-after-10min' (time budget 600s exceeded)\",\n \"status\": 0\n }\n ],\n \"next\": null,\n \"previous\": null\n }\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "org", + "events" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantEvents\",\n \"Data\": {\n \"auditEvents\": [\n {\n \"id\": \"a6f1e62-3456-4cde-9def-3c4d5e6f7g8h\",\n \"createdOn\": \"2026-05-19T08:10:00.105Z\",\n \"organizationId\": \"11111111-2222-3333-4444-555555555555\",\n \"organizationName\": \"Acme Corp\",\n \"tenantId\": \"22222222-3333-4444-5555-666666666666\",\n \"tenantName\": \"Acme-Prod\",\n \"actorId\": \"00000000-0000-0000-0000-000000000000\",\n \"actorName\": \"System\",\n \"actorEmail\": null,\n \"eventType\": \"Job stopped\",\n \"eventSource\": \"Orchestrator\",\n \"eventTarget\": \"Job\",\n \"eventDetails\": \"{\\\"jobKey\\\":\\\"c3c0b2c3-d4e5-4678-9012-3456789ccccc\\\",\\\"processName\\\":\\\"LongRunningProcess\\\",\\\"strategy\\\":\\\"Kill\\\",\\\"origin\\\":\\\"Trigger\\\",\\\"folderKey\\\":\\\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\\\",\\\"folderName\\\":\\\"KillScenarios\\\",\\\"actorType\\\":\\\"System\\\",\\\"triggerName\\\":\\\"kill-after-10min\\\",\\\"triggerType\\\":\\\"AutoCancel\\\",\\\"timeBudgetSeconds\\\":600,\\\"reason\\\":\\\"TimeBudgetExceeded\\\"}\",\n \"eventSummary\": \"Job 'LongRunningProcess' (c3c0b2c3-d4e5-4678-9012-3456789ccccc) was stopped (strategy Kill) by auto-cancel trigger 'kill-after-10min' (time budget 600s exceeded)\",\n \"status\": 0\n }\n ],\n \"next\": null,\n \"previous\": null\n }\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "tenant", + "sources" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantSources\",\n \"Data\": [\n {\n \"id\": \"33333333-aaaa-4444-bbbb-555555555555\",\n \"name\": \"Orchestrator\",\n \"targets\": [\n {\n \"id\": \"44444444-cccc-5555-dddd-666666666666\",\n \"name\": \"Job\"\n }\n ],\n \"types\": [\n {\n \"id\": \"55555555-eeee-6666-ffff-777777777777\",\n \"name\": \"Job stopped\"\n }\n ]\n }\n ]\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "org", + "sources" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantSources\",\n \"Data\": [\n {\n \"id\": \"33333333-aaaa-4444-bbbb-555555555555\",\n \"name\": \"Orchestrator\",\n \"targets\": [\n {\n \"id\": \"44444444-cccc-5555-dddd-666666666666\",\n \"name\": \"Job\"\n }\n ],\n \"types\": [\n {\n \"id\": \"55555555-eeee-6666-ffff-777777777777\",\n \"name\": \"Job stopped\"\n }\n ]\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Name\": \"KillScenarios\",\r\n \"Path\": \"KillScenarios\",\r\n \"Description\": \"Folder hosting LongRunningProcess — long-running job used to exercise the 0x40010004 (Job killed) playbook variants.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs get c3c0b2c3", + "min": 1, + "description": "Agent must inspect the failing job to confirm the exit code" + }, + { + "pattern": "or jobs history c3c0b2c3", + "min": 1, + "description": "Agent must inspect state history to distinguish operator-initiated Stop from host-side termination" + }, + { + "pattern": "admin audit", + "min": 1, + "description": "Agent must query the audit log to identify the actor" + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/task.yaml index 50e9edb898..08d731c60c 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-autocancel-trigger/task.yaml @@ -24,21 +24,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/README.md new file mode 100644 index 0000000000..5664c5ac84 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/README.md @@ -0,0 +1,71 @@ +# Rpa Job Killed By Operator UI — Force-Kill From Orchestrator + +This scenario reproduces a runtime fault caused by **a human operator +force-killing a running Orchestrator job via the UI**. The job ends +with: + +``` +System.Exception: Job stopped with an unexpected exit code: 0x40010004 +``` + +(Windows `DBG_TERMINATE_PROCESS`.) + +## What this scenario uncovers + +**Root Cause:** Job `LongRunningProcess` (key `a1c0b2c3-...`) was +running normally when an operator clicked **Stop → Kill** in +Orchestrator. The Kill strategy calls `TerminateProcess` on the +executor, which Windows reports as exit code `0x40010004`. The audit +log identifies the actor: `Jane Doe (jane.doe@acme.com)` issued the +stop with strategy `Kill` via the Orchestrator UI. + +This maps to: +`references/products/orchestrator/playbooks/job-stopped-exit-code-0x40010004.md` +(the "Operator Kill from Orchestrator" branch). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal `LongRunningProcess` project — single 5-minute Delay activity, just enough to make the job long-running and killable mid-execution | + +> **Note on fixtures.** Like sibling synthetic scenarios, the fixtures +> here were authored from the documented playbook signature rather +> than captured from a real `.investigation/` session. The audit +> event content names a synthetic actor (Jane Doe) — the test grades +> whether the agent surfaces the actor identity, not the specific +> name. + +## How this differs from the sibling Kill variants + +There are **three** scenarios that all exercise the +`job-stopped-exit-code-0x40010004` playbook's Operator Kill branch, +each with a different actor surfaced in the audit event: + +| Scenario | Actor | Surface signal | +|---|---|---| +| **this** — `rpa-job-killed-by-operator-ui` | Human user via Orchestrator UI | `actorEmail` is a real user; `clientInfo.ipAddress` present | +| `rpa-job-killed-by-watchdog-account` | Robot / automation account via API | `actorName` is a service account; `eventSummary` mentions "via API" | +| `rpa-job-killed-by-autocancel-trigger` | System-initiated cancellation trigger | `actorName` is `System`; `eventDetails` references the trigger name and time-budget | + +A correct agent matches the **0x40010004 playbook → Operator Kill +branch** and identifies the actor from the audit log. The branch +selection is the same across all three; the differentiator is +**who** the agent names in its conclusion. + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `orchestrator/job-stopped-exit-code-0x40010004` +- Agent ran `uip or jobs history` and observed the + `Running → Stopping → Faulted` transition (operator-initiated) +- Agent ran `uip admin audit tenant events --search ` and + named the actor (Jane Doe / jane.doe@acme.com) in the conclusion +- Conclusion recommends a fix from the acceptable list (SoftStop vs + Kill training, restrict `Jobs.Edit`, or confirm intent) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/0dac23d736.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/0dac23d736.json deleted file mode 100644 index 39c682ad6e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/0dac23d736.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Faulted-only view: just the failing LongRunningProcess job.", - "Data": [ - { - "Key": "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", - "State": "Faulted", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:01:32.105Z", - "CreationTime": "2026-05-19T08:00:00.302Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/34e5b0c520.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/34e5b0c520.json deleted file mode 100644 index f5b8165f3f..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/34e5b0c520.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "AuditTenantEvents", - "_doc": "Audit query returns the single Job.Stop event matching the failing job key. The actor is a named human user — jane.doe@acme.com — who issued Kill via the Orchestrator UI from a known IP. This is the operator-initiated-Kill signature.", - "Data": { - "auditEvents": [ - { - "id": "8d4f1e62-1234-4abc-9def-1a2b3c4d5e6f", - "createdOn": "2026-05-19T08:01:30.105Z", - "organizationId": "11111111-2222-3333-4444-555555555555", - "organizationName": "Acme Corp", - "tenantId": "22222222-3333-4444-5555-666666666666", - "tenantName": "Acme-Prod", - "actorId": "aaaa1111-bbbb-2222-cccc-3333dddd4444", - "actorName": "Jane Doe", - "actorEmail": "jane.doe@acme.com", - "eventType": "Job stopped", - "eventSource": "Orchestrator", - "eventTarget": "Job", - "eventDetails": "{\"jobKey\":\"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\",\"processName\":\"LongRunningProcess\",\"strategy\":\"Kill\",\"origin\":\"OrchestratorUI\",\"folderKey\":\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\"folderName\":\"KillScenarios\"}", - "eventSummary": "Job 'LongRunningProcess' (a1c0b2c3-d4e5-4678-9012-3456789aaaaa) was stopped (strategy Kill) by jane.doe@acme.com via Orchestrator UI", - "status": 0, - "clientInfo": { - "ipAddress": "10.50.12.34", - "ipCountry": "US" - } - } - ], - "next": null, - "previous": null - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/63044a0ebf.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/63044a0ebf.json deleted file mode 100644 index 7728fc0207..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/63044a0ebf.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Name": "KillScenarios", - "Path": "KillScenarios", - "Description": "Folder hosting LongRunningProcess — long-running job used to exercise the 0x40010004 (Job killed) playbook variants.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/7cb37c3120.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/7cb37c3120.json deleted file mode 100644 index 5b7278cc01..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/7cb37c3120.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "_doc": "State transitions for the failing job. The presence of 'Stopping' is the signal that the kill was operator-initiated from Orchestrator. Host-side terminations (logoff, OOM, service restart) would go Running → Faulted directly with no Stopping/Killing.", - "Data": [ - { - "State": "Pending", - "Timestamp": "2026-05-19T08:00:00.302Z", - "Reason": "JobCreated" - }, - { - "State": "Running", - "Timestamp": "2026-05-19T08:00:00.500Z", - "Reason": "ExecutorStarted" - }, - { - "State": "Stopping", - "Timestamp": "2026-05-19T08:01:30.000Z", - "Reason": "StopRequested", - "Strategy": "Kill" - }, - { - "State": "Faulted", - "Timestamp": "2026-05-19T08:01:32.105Z", - "Reason": "ExecutorExited", - "ExitCode": "0x40010004" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/847a2b86cb.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/847a2b86cb.json deleted file mode 100644 index 69f59929bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/847a2b86cb.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:01:32.105Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "7d2e7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f", - "Info": "System.Exception: Job stopped with an unexpected exit code: 0x40010004\r\n at UiPath.Robot.Service.Hosting.ExecutorSupervisor.HandleExitAsync(Int32 exitCode, CancellationToken cancellationToken)\r\n at UiPath.Robot.Service.JobManager.FinalizeJobAsync(JobRecord record, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-19T08:00:00.302Z", - "ReleaseName": "LongRunningProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5670099, - "OrganizationUnitId": 7791617, - "FolderKey": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:01:32.105Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", - "Id": 670911011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/README.md deleted file mode 100644 index c40116d0c3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/README.md +++ /dev/null @@ -1,70 +0,0 @@ -# Rpa Job Killed By Operator UI — Force-Kill From Orchestrator - -This scenario reproduces a runtime fault caused by **a human operator -force-killing a running Orchestrator job via the UI**. The job ends -with: - -``` -System.Exception: Job stopped with an unexpected exit code: 0x40010004 -``` - -(Windows `DBG_TERMINATE_PROCESS`.) - -## What this scenario uncovers - -**Root Cause:** Job `LongRunningProcess` (key `a1c0b2c3-...`) was -running normally when an operator clicked **Stop → Kill** in -Orchestrator. The Kill strategy calls `TerminateProcess` on the -executor, which Windows reports as exit code `0x40010004`. The audit -log identifies the actor: `Jane Doe (jane.doe@acme.com)` issued the -stop with strategy `Kill` via the Orchestrator UI. - -This maps to: -`references/products/orchestrator/playbooks/job-stopped-exit-code-0x40010004.md` -(the "Operator Kill from Orchestrator" branch). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | minimal `LongRunningProcess` project — single 5-minute Delay activity, just enough to make the job long-running and killable mid-execution | -| `fixtures/mocks/responses/*.json` | **synthetic** canned `uip` responses authored from the documented playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** Like sibling synthetic scenarios, the fixtures -> here were authored from the documented playbook signature rather -> than captured from a real `.investigation/` session. The audit -> event content names a synthetic actor (Jane Doe) — the test grades -> whether the agent surfaces the actor identity, not the specific -> name. - -## How this differs from the sibling Kill variants - -There are **three** scenarios that all exercise the -`job-stopped-exit-code-0x40010004` playbook's Operator Kill branch, -each with a different actor surfaced in the audit event: - -| Scenario | Actor | Surface signal | -|---|---|---| -| **this** — `rpa-job-killed-by-operator-ui` | Human user via Orchestrator UI | `actorEmail` is a real user; `clientInfo.ipAddress` present | -| `rpa-job-killed-by-watchdog-account` | Robot / automation account via API | `actorName` is a service account; `eventSummary` mentions "via API" | -| `rpa-job-killed-by-autocancel-trigger` | System-initiated cancellation trigger | `actorName` is `System`; `eventDetails` references the trigger name and time-budget | - -A correct agent matches the **0x40010004 playbook → Operator Kill -branch** and identifies the actor from the audit log. The branch -selection is the same across all three; the differentiator is -**who** the agent names in its conclusion. - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `orchestrator/job-stopped-exit-code-0x40010004` -- Agent ran `uip or jobs history` and observed the - `Running → Stopping → Faulted` transition (operator-initiated) -- Agent ran `uip admin audit tenant events --search ` and - named the actor (Jane Doe / jane.doe@acme.com) in the conclusion -- Conclusion recommends a fix from the acceptable list (SoftStop vs - Kill training, restrict `Jobs.Edit`, or confirm intent) diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/be1ccc7600.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/be1ccc7600.json deleted file mode 100644 index fcf2f805db..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/be1ccc7600.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "AuditTenantSources", - "_doc": "Minimal sources discovery payload. Real responses include many more sources; this stub lists only the ones relevant to job-stop investigations so the agent has plausible GUIDs if it filters by --source.", - "Data": [ - { - "id": "33333333-aaaa-4444-bbbb-555555555555", - "name": "Orchestrator", - "targets": [ - { - "id": "44444444-cccc-5555-dddd-666666666666", - "name": "Job" - } - ], - "types": [ - { - "id": "55555555-eeee-6666-ffff-777777777777", - "name": "Job stopped" - } - ] - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/ea44e4b8c4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/ea44e4b8c4.json deleted file mode 100644 index 3810b83adc..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/ea44e4b8c4.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent jobs in KillScenarios. The failing job (a1c0b2c3) is preceded by several successful LongRunningProcess runs — no overlap, no other Killing state — so the kill is not a recurring pattern.", - "Data": [ - { - "Key": "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", - "State": "Faulted", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:01:32.105Z", - "CreationTime": "2026-05-19T08:00:00.302Z" - }, - { - "Key": "9aa0b2c3-d4e5-4678-9012-3456789aaaa1", - "State": "Successful", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T08:00:00.500Z", - "EndTime": "2026-05-18T08:05:00.812Z", - "CreationTime": "2026-05-18T08:00:00.295Z" - }, - { - "Key": "8aa0b2c3-d4e5-4678-9012-3456789aaaa2", - "State": "Successful", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-17T08:00:00.500Z", - "EndTime": "2026-05-17T08:05:00.755Z", - "CreationTime": "2026-05-17T08:00:00.310Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/f1f8f2e0f9.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/f1f8f2e0f9.json deleted file mode 100644 index 4c7709db6f..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/f1f8f2e0f9.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "_doc": "Execution logs end abruptly mid-Delay — no Job Completed line. The last user-workflow message is the start of the long-running step. The Robot Service then logs the kill request and the abrupt exit.", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-19T08:01:32.105Z", - "Message": "Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed." - }, - { - "Level": "Warn", - "TimeStamp": "2026-05-19T08:01:30.005Z", - "Message": "[Robot] Stop request received for job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (strategy=Kill). Terminating executor process." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.520Z", - "Message": "[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.510Z", - "Message": "Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.310Z", - "Message": "Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' created. Source=Manual, ReleaseName=LongRunningProcess, RequiresUserInteraction=False." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/manifest.json deleted file mode 100644 index edf6e247f3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/m/r/manifest.json +++ /dev/null @@ -1,93 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-job-killed-by-operator-ui scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "63044a0ebf.json" - }, - { - "match": "or folders list --output json", - "file": "63044a0ebf.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --state Faulted --output json", - "file": "0dac23d736.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --state Faulted --output json", - "file": "0dac23d736.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --output json", - "file": "ea44e4b8c4.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --output json", - "file": "ea44e4b8c4.json" - }, - { - "match": "or jobs get \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\" --output json", - "file": "847a2b86cb.json" - }, - { - "match": "or jobs get a1c0b2c3-d4e5-4678-9012-3456789aaaaa --output json", - "file": "847a2b86cb.json" - }, - { - "match": "or jobs history \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\" --output json", - "file": "7cb37c3120.json" - }, - { - "match": "or jobs history a1c0b2c3-d4e5-4678-9012-3456789aaaaa --output json", - "file": "7cb37c3120.json" - }, - { - "match": "or jobs logs \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\" --level Error --output json", - "file": "f1f8f2e0f9.json" - }, - { - "match": "or jobs logs a1c0b2c3-d4e5-4678-9012-3456789aaaaa --level Error --output json", - "file": "f1f8f2e0f9.json" - }, - { - "match": "or jobs logs \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\" --output json", - "file": "f1f8f2e0f9.json" - }, - { - "match": "or jobs logs a1c0b2c3-d4e5-4678-9012-3456789aaaaa --output json", - "file": "f1f8f2e0f9.json" - }, - { - "match": "admin audit tenant events", - "file": "34e5b0c520.json" - }, - { - "match": "admin audit org events", - "file": "34e5b0c520.json" - }, - { - "match": "admin audit tenant sources", - "file": "be1ccc7600.json" - }, - { - "match": "admin audit org sources", - "file": "be1ccc7600.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { "pattern": "or jobs get a1c0b2c3", "min": 1, "description": "Agent must inspect the failing job to confirm the exit code" }, - { "pattern": "or jobs history a1c0b2c3", "min": 1, "description": "Agent must inspect state history to distinguish operator-initiated Stop from host-side termination" }, - { "pattern": "admin audit", "min": 1, "description": "Agent must query the audit log to identify the actor" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/uip-fixture.json new file mode 100644 index 0000000000..1b08a67e7d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/data/uip-fixture.json @@ -0,0 +1,166 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\n \"EndTime\": \"2026-05-19T08:01:32.105Z\",\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobLogs\",\n \"Data\": [\n {\n \"Level\": \"Error\",\n \"TimeStamp\": \"2026-05-19T08:01:32.105Z\",\n \"Message\": \"Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed.\"\n },\n {\n \"Level\": \"Warn\",\n \"TimeStamp\": \"2026-05-19T08:01:30.005Z\",\n \"Message\": \"[Robot] Stop request received for job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (strategy=Kill). Terminating executor process.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.520Z\",\n \"Message\": \"[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.510Z\",\n \"Message\": \"Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.310Z\",\n \"Message\": \"Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' created. Source=Manual, ReleaseName=LongRunningProcess, RequiresUserInteraction=False.\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\n \"EndTime\": \"2026-05-19T08:01:32.105Z\",\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\"\n },\n {\n \"Key\": \"9aa0b2c3-d4e5-4678-9012-3456789aaaa1\",\n \"State\": \"Successful\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-18T08:00:00.500Z\",\n \"EndTime\": \"2026-05-18T08:05:00.812Z\",\n \"CreationTime\": \"2026-05-18T08:00:00.295Z\"\n },\n {\n \"Key\": \"8aa0b2c3-d4e5-4678-9012-3456789aaaa2\",\n \"State\": \"Successful\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-17T08:00:00.500Z\",\n \"EndTime\": \"2026-05-17T08:05:00.755Z\",\n \"CreationTime\": \"2026-05-17T08:00:00.310Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\",\r\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T08:01:32.105Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"7d2e7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f\",\r\n \"Info\": \"System.Exception: Job stopped with an unexpected exit code: 0x40010004\\r\\n at UiPath.Robot.Service.Hosting.ExecutorSupervisor.HandleExitAsync(Int32 exitCode, CancellationToken cancellationToken)\\r\\n at UiPath.Robot.Service.JobManager.FinalizeJobAsync(JobRecord record, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\",\r\n \"ReleaseName\": \"LongRunningProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5670099,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:01:32.105Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\",\r\n \"Id\": 670911011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobHistory\",\n \"Data\": [\n {\n \"State\": \"Pending\",\n \"Timestamp\": \"2026-05-19T08:00:00.302Z\",\n \"Reason\": \"JobCreated\"\n },\n {\n \"State\": \"Running\",\n \"Timestamp\": \"2026-05-19T08:00:00.500Z\",\n \"Reason\": \"ExecutorStarted\"\n },\n {\n \"State\": \"Stopping\",\n \"Timestamp\": \"2026-05-19T08:01:30.000Z\",\n \"Reason\": \"StopRequested\",\n \"Strategy\": \"Kill\"\n },\n {\n \"State\": \"Faulted\",\n \"Timestamp\": \"2026-05-19T08:01:32.105Z\",\n \"Reason\": \"ExecutorExited\",\n \"ExitCode\": \"0x40010004\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1c0b2c3-d4e5-4678-9012-3456789aaaaa", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobLogs\",\n \"Data\": [\n {\n \"Level\": \"Error\",\n \"TimeStamp\": \"2026-05-19T08:01:32.105Z\",\n \"Message\": \"Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed.\"\n },\n {\n \"Level\": \"Warn\",\n \"TimeStamp\": \"2026-05-19T08:01:30.005Z\",\n \"Message\": \"[Robot] Stop request received for job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (strategy=Kill). Terminating executor process.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.520Z\",\n \"Message\": \"[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.510Z\",\n \"Message\": \"Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.310Z\",\n \"Message\": \"Job 'a1c0b2c3-d4e5-4678-9012-3456789aaaaa' created. Source=Manual, ReleaseName=LongRunningProcess, RequiresUserInteraction=False.\"\n }\n ]\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "tenant", + "events" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantEvents\",\n \"Data\": {\n \"auditEvents\": [\n {\n \"id\": \"8d4f1e62-1234-4abc-9def-1a2b3c4d5e6f\",\n \"createdOn\": \"2026-05-19T08:01:30.105Z\",\n \"organizationId\": \"11111111-2222-3333-4444-555555555555\",\n \"organizationName\": \"Acme Corp\",\n \"tenantId\": \"22222222-3333-4444-5555-666666666666\",\n \"tenantName\": \"Acme-Prod\",\n \"actorId\": \"aaaa1111-bbbb-2222-cccc-3333dddd4444\",\n \"actorName\": \"Jane Doe\",\n \"actorEmail\": \"jane.doe@acme.com\",\n \"eventType\": \"Job stopped\",\n \"eventSource\": \"Orchestrator\",\n \"eventTarget\": \"Job\",\n \"eventDetails\": \"{\\\"jobKey\\\":\\\"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\\\",\\\"processName\\\":\\\"LongRunningProcess\\\",\\\"strategy\\\":\\\"Kill\\\",\\\"origin\\\":\\\"OrchestratorUI\\\",\\\"folderKey\\\":\\\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\\\",\\\"folderName\\\":\\\"KillScenarios\\\"}\",\n \"eventSummary\": \"Job 'LongRunningProcess' (a1c0b2c3-d4e5-4678-9012-3456789aaaaa) was stopped (strategy Kill) by jane.doe@acme.com via Orchestrator UI\",\n \"status\": 0,\n \"clientInfo\": {\n \"ipAddress\": \"10.50.12.34\",\n \"ipCountry\": \"US\"\n }\n }\n ],\n \"next\": null,\n \"previous\": null\n }\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "org", + "events" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantEvents\",\n \"Data\": {\n \"auditEvents\": [\n {\n \"id\": \"8d4f1e62-1234-4abc-9def-1a2b3c4d5e6f\",\n \"createdOn\": \"2026-05-19T08:01:30.105Z\",\n \"organizationId\": \"11111111-2222-3333-4444-555555555555\",\n \"organizationName\": \"Acme Corp\",\n \"tenantId\": \"22222222-3333-4444-5555-666666666666\",\n \"tenantName\": \"Acme-Prod\",\n \"actorId\": \"aaaa1111-bbbb-2222-cccc-3333dddd4444\",\n \"actorName\": \"Jane Doe\",\n \"actorEmail\": \"jane.doe@acme.com\",\n \"eventType\": \"Job stopped\",\n \"eventSource\": \"Orchestrator\",\n \"eventTarget\": \"Job\",\n \"eventDetails\": \"{\\\"jobKey\\\":\\\"a1c0b2c3-d4e5-4678-9012-3456789aaaaa\\\",\\\"processName\\\":\\\"LongRunningProcess\\\",\\\"strategy\\\":\\\"Kill\\\",\\\"origin\\\":\\\"OrchestratorUI\\\",\\\"folderKey\\\":\\\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\\\",\\\"folderName\\\":\\\"KillScenarios\\\"}\",\n \"eventSummary\": \"Job 'LongRunningProcess' (a1c0b2c3-d4e5-4678-9012-3456789aaaaa) was stopped (strategy Kill) by jane.doe@acme.com via Orchestrator UI\",\n \"status\": 0,\n \"clientInfo\": {\n \"ipAddress\": \"10.50.12.34\",\n \"ipCountry\": \"US\"\n }\n }\n ],\n \"next\": null,\n \"previous\": null\n }\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "tenant", + "sources" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantSources\",\n \"Data\": [\n {\n \"id\": \"33333333-aaaa-4444-bbbb-555555555555\",\n \"name\": \"Orchestrator\",\n \"targets\": [\n {\n \"id\": \"44444444-cccc-5555-dddd-666666666666\",\n \"name\": \"Job\"\n }\n ],\n \"types\": [\n {\n \"id\": \"55555555-eeee-6666-ffff-777777777777\",\n \"name\": \"Job stopped\"\n }\n ]\n }\n ]\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "org", + "sources" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantSources\",\n \"Data\": [\n {\n \"id\": \"33333333-aaaa-4444-bbbb-555555555555\",\n \"name\": \"Orchestrator\",\n \"targets\": [\n {\n \"id\": \"44444444-cccc-5555-dddd-666666666666\",\n \"name\": \"Job\"\n }\n ],\n \"types\": [\n {\n \"id\": \"55555555-eeee-6666-ffff-777777777777\",\n \"name\": \"Job stopped\"\n }\n ]\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Name\": \"KillScenarios\",\r\n \"Path\": \"KillScenarios\",\r\n \"Description\": \"Folder hosting LongRunningProcess — long-running job used to exercise the 0x40010004 (Job killed) playbook variants.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs get a1c0b2c3", + "min": 1, + "description": "Agent must inspect the failing job to confirm the exit code" + }, + { + "pattern": "or jobs history a1c0b2c3", + "min": 1, + "description": "Agent must inspect state history to distinguish operator-initiated Stop from host-side termination" + }, + { + "pattern": "admin audit", + "min": 1, + "description": "Agent must query the audit log to identify the actor" + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/task.yaml index e1ade97b3a..64de0d951f 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-operator-ui/task.yaml @@ -22,21 +22,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/README.md new file mode 100644 index 0000000000..4cf0a9950c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/README.md @@ -0,0 +1,69 @@ +# Rpa Job Killed By Watchdog Account — API Kill From Service Account + +This scenario reproduces a runtime fault caused by **a watchdog +automation account force-killing a running Orchestrator job via the +REST API**. The job ends with: + +``` +System.Exception: Job stopped with an unexpected exit code: 0x40010004 +``` + +(Windows `DBG_TERMINATE_PROCESS`.) + +## What this scenario uncovers + +**Root Cause:** Job `LongRunningProcess` (key `b2c0b2c3-...`) was +running when an automated watchdog (`watchdog-svc@acme.com`) issued +`POST /jobs/{key}/stop` with `strategy=Kill` via the REST API. The +watchdog rule (`HungJobReaper`) had a 180-second threshold and +flagged the job after it exceeded that threshold during a slow +inference. The audit log identifies the actor as a non-human +service account (no `clientInfo` block — there is no browser +session for an API call). + +This maps to: +`references/products/orchestrator/playbooks/job-stopped-exit-code-0x40010004.md` +(the "Operator Kill from Orchestrator" branch, API origin). + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal `LongRunningProcess` project — single 5-minute Delay activity | + +The audit event's `actorName` (`watchdog-svc`), absence of +`clientInfo`, `origin: "API"` in `eventDetails`, and the +`watchdogRule` / `watchdogThresholdSeconds` fields together +identify a service-account-driven kill rather than a UI click. + +## How this differs from the sibling Kill variants + +| Scenario | Actor | Surface signal | +|---|---|---| +| `rpa-job-killed-by-operator-ui` | Human user via Orchestrator UI | `actorEmail` is a real user; `clientInfo.ipAddress` present | +| **this** — `rpa-job-killed-by-watchdog-account` | Service account via API | `actorName` is non-human (`watchdog-svc`); `clientInfo` absent; `eventDetails.origin=API` | +| `rpa-job-killed-by-autocancel-trigger` | System-initiated cancellation trigger | `actorName` is `System`; `eventDetails` references trigger name and time budget | + +A correct agent matches the **0x40010004 playbook → Operator Kill +branch** and recognizes that the actor is **not a human** by +inspecting the audit fields (no `clientInfo`, non-human +`actorName`, `origin: "API"`). + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched `orchestrator/job-stopped-exit-code-0x40010004` +- Agent ran `uip or jobs history` and observed the + `Running → Killing → Faulted` transition +- Agent ran `uip admin audit tenant events --search ` and + identified the actor as a service account (named `watchdog-svc` + or equivalent), recognizing API origin and absence of `clientInfo` +- Conclusion recommends a fix from the acceptable list (review the + watchdog rule threshold, restrict watchdog API credentials, or + accept the kill if it was legitimate) + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/058bd748f1.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/058bd748f1.json deleted file mode 100644 index 783bc38553..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/058bd748f1.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Faulted-only view: just the failing LongRunningProcess job.", - "Data": [ - { - "Key": "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", - "State": "Faulted", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Manual", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:03:01.255Z", - "CreationTime": "2026-05-19T08:00:00.302Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/0a55b81a39.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/0a55b81a39.json deleted file mode 100644 index fadc071089..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/0a55b81a39.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "_doc": "Recent jobs in KillScenarios. The failing job (b2c0b2c3) is preceded by several successful LongRunningProcess runs — no overlap, no other Killing state — so the kill is not a recurring pattern.", - "Data": [ - { - "Key": "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", - "State": "Faulted", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:03:01.255Z", - "CreationTime": "2026-05-19T08:00:00.302Z" - }, - { - "Key": "9aa0b2c3-d4e5-4678-9012-3456789aaaa1", - "State": "Successful", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-18T08:00:00.500Z", - "EndTime": "2026-05-18T08:05:00.812Z", - "CreationTime": "2026-05-18T08:00:00.295Z" - }, - { - "Key": "8aa0b2c3-d4e5-4678-9012-3456789aaaa2", - "State": "Successful", - "ProcessName": "LongRunningProcess", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "MachineKey": "", - "StartTime": "2026-05-17T08:00:00.500Z", - "EndTime": "2026-05-17T08:05:00.755Z", - "CreationTime": "2026-05-17T08:00:00.310Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/2be52d8ea7.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/2be52d8ea7.json deleted file mode 100644 index 73fbd0936c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/2be52d8ea7.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", - "StartTime": "2026-05-19T08:00:00.500Z", - "EndTime": "2026-05-19T08:03:01.255Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "8e3f7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f", - "Info": "System.Exception: Job stopped with an unexpected exit code: 0x40010004\r\n at UiPath.Robot.Service.Hosting.ExecutorSupervisor.HandleExitAsync(Int32 exitCode, CancellationToken cancellationToken)\r\n at UiPath.Robot.Service.JobManager.FinalizeJobAsync(JobRecord record, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-05-19T08:00:00.302Z", - "ReleaseName": "LongRunningProcess", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5670099, - "OrganizationUnitId": 7791617, - "FolderKey": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\AUTOMATION1", - "OrchestratorUserIdentity": "automation1", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-19T08:03:01.255Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", - "Id": 670911044 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/5fb830e5b4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/5fb830e5b4.json deleted file mode 100644 index 653263dca0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/5fb830e5b4.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "AuditTenantEvents", - "_doc": "Audit query returns the Job.Stop event matching the failing job key. The actor is a non-human service account — watchdog-svc@acme.com — that issued Kill via the REST API. clientInfo is absent (no browser session). eventDetails references a watchdog rule name 'HungJobReaper'.", - "Data": { - "auditEvents": [ - { - "id": "9e5f1e62-2345-4bcd-9def-2b3c4d5e6f7g", - "createdOn": "2026-05-19T08:03:00.105Z", - "organizationId": "11111111-2222-3333-4444-555555555555", - "organizationName": "Acme Corp", - "tenantId": "22222222-3333-4444-5555-666666666666", - "tenantName": "Acme-Prod", - "actorId": "bbbb2222-cccc-3333-dddd-4444eeee5555", - "actorName": "watchdog-svc", - "actorEmail": "watchdog-svc@acme.com", - "eventType": "Job stopped", - "eventSource": "Orchestrator", - "eventTarget": "Job", - "eventDetails": "{\"jobKey\":\"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\",\"processName\":\"LongRunningProcess\",\"strategy\":\"Kill\",\"origin\":\"API\",\"folderKey\":\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\"folderName\":\"KillScenarios\",\"actorType\":\"ServiceAccount\",\"watchdogRule\":\"HungJobReaper\",\"watchdogThresholdSeconds\":180,\"clientApp\":\"watchdog-svc/1.4.2\"}", - "eventSummary": "Job 'LongRunningProcess' (b2c0b2c3-d4e5-4678-9012-3456789bbbbb) was stopped (strategy Kill) by service account watchdog-svc@acme.com via API (rule: HungJobReaper, threshold: 180s)", - "status": 0 - } - ], - "next": null, - "previous": null - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/63044a0ebf.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/63044a0ebf.json deleted file mode 100644 index 7728fc0207..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/63044a0ebf.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "f1e2d3c4-b5a6-7890-1234-56789abcdef0", - "Name": "KillScenarios", - "Path": "KillScenarios", - "Description": "Folder hosting LongRunningProcess — long-running job used to exercise the 0x40010004 (Job killed) playbook variants.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/README.md deleted file mode 100644 index 1a54590a40..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/README.md +++ /dev/null @@ -1,68 +0,0 @@ -# Rpa Job Killed By Watchdog Account — API Kill From Service Account - -This scenario reproduces a runtime fault caused by **a watchdog -automation account force-killing a running Orchestrator job via the -REST API**. The job ends with: - -``` -System.Exception: Job stopped with an unexpected exit code: 0x40010004 -``` - -(Windows `DBG_TERMINATE_PROCESS`.) - -## What this scenario uncovers - -**Root Cause:** Job `LongRunningProcess` (key `b2c0b2c3-...`) was -running when an automated watchdog (`watchdog-svc@acme.com`) issued -`POST /jobs/{key}/stop` with `strategy=Kill` via the REST API. The -watchdog rule (`HungJobReaper`) had a 180-second threshold and -flagged the job after it exceeded that threshold during a slow -inference. The audit log identifies the actor as a non-human -service account (no `clientInfo` block — there is no browser -session for an API call). - -This maps to: -`references/products/orchestrator/playbooks/job-stopped-exit-code-0x40010004.md` -(the "Operator Kill from Orchestrator" branch, API origin). - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` | -| `process/` | minimal `LongRunningProcess` project — single 5-minute Delay activity | -| `fixtures/mocks/responses/*.json` | synthetic `uip` responses authored from the playbook signature | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -The audit event's `actorName` (`watchdog-svc`), absence of -`clientInfo`, `origin: "API"` in `eventDetails`, and the -`watchdogRule` / `watchdogThresholdSeconds` fields together -identify a service-account-driven kill rather than a UI click. - -## How this differs from the sibling Kill variants - -| Scenario | Actor | Surface signal | -|---|---|---| -| `rpa-job-killed-by-operator-ui` | Human user via Orchestrator UI | `actorEmail` is a real user; `clientInfo.ipAddress` present | -| **this** — `rpa-job-killed-by-watchdog-account` | Service account via API | `actorName` is non-human (`watchdog-svc`); `clientInfo` absent; `eventDetails.origin=API` | -| `rpa-job-killed-by-autocancel-trigger` | System-initiated cancellation trigger | `actorName` is `System`; `eventDetails` references trigger name and time budget | - -A correct agent matches the **0x40010004 playbook → Operator Kill -branch** and recognizes that the actor is **not a human** by -inspecting the audit fields (no `clientInfo`, non-human -`actorName`, `origin: "API"`). - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched `orchestrator/job-stopped-exit-code-0x40010004` -- Agent ran `uip or jobs history` and observed the - `Running → Killing → Faulted` transition -- Agent ran `uip admin audit tenant events --search ` and - identified the actor as a service account (named `watchdog-svc` - or equivalent), recognizing API origin and absence of `clientInfo` -- Conclusion recommends a fix from the acceptable list (review the - watchdog rule threshold, restrict watchdog API credentials, or - accept the kill if it was legitimate) diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/b420de190c.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/b420de190c.json deleted file mode 100644 index 4754421012..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/b420de190c.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "_doc": "Execution logs end abruptly mid-Delay — no Job Completed line. Last user-workflow line is the start of the long-running step. The Robot Service logs the kill request and abrupt exit. Kill request source is recorded as API.", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-19T08:03:01.255Z", - "Message": "Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed." - }, - { - "Level": "Warn", - "TimeStamp": "2026-05-19T08:03:00.005Z", - "Message": "[Robot] Stop request received for job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (strategy=Kill, source=API). Terminating executor process." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.520Z", - "Message": "[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.510Z", - "Message": "Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST." - }, - { - "Level": "Info", - "TimeStamp": "2026-05-19T08:00:00.310Z", - "Message": "Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' created. Source=Schedule, ReleaseName=LongRunningProcess, RequiresUserInteraction=False." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/be1ccc7600.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/be1ccc7600.json deleted file mode 100644 index fcf2f805db..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/be1ccc7600.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "Result": "Success", - "Code": "AuditTenantSources", - "_doc": "Minimal sources discovery payload. Real responses include many more sources; this stub lists only the ones relevant to job-stop investigations so the agent has plausible GUIDs if it filters by --source.", - "Data": [ - { - "id": "33333333-aaaa-4444-bbbb-555555555555", - "name": "Orchestrator", - "targets": [ - { - "id": "44444444-cccc-5555-dddd-666666666666", - "name": "Job" - } - ], - "types": [ - { - "id": "55555555-eeee-6666-ffff-777777777777", - "name": "Job stopped" - } - ] - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/f27ee3cf71.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/f27ee3cf71.json deleted file mode 100644 index ff1ec1696b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/f27ee3cf71.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "_doc": "State transitions for the failing job. The presence of 'Killing' confirms an operator-initiated Kill — distinguishes from host-side termination (which goes Running → Faulted directly). The transition into Killing was triggered by the REST API (Source=API on the transition reason).", - "Data": [ - { - "State": "Pending", - "Timestamp": "2026-05-19T08:00:00.302Z", - "Reason": "JobCreated" - }, - { - "State": "Running", - "Timestamp": "2026-05-19T08:00:00.500Z", - "Reason": "ExecutorStarted" - }, - { - "State": "Killing", - "Timestamp": "2026-05-19T08:03:00.000Z", - "Reason": "StopRequested", - "Strategy": "Kill", - "Source": "API" - }, - { - "State": "Faulted", - "Timestamp": "2026-05-19T08:03:01.255Z", - "Reason": "ExecutorExited", - "ExitCode": "0x40010004" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/manifest.json deleted file mode 100644 index 3f265edd75..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/m/r/manifest.json +++ /dev/null @@ -1,93 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-job-killed-by-watchdog-account scenario; mock dispatcher only — do not treat as investigation evidence.", - "rules": [ - { - "match": "or folders list --output json", - "file": "63044a0ebf.json" - }, - { - "match": "or folders list --output json", - "file": "63044a0ebf.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --state Faulted --output json", - "file": "058bd748f1.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --state Faulted --output json", - "file": "058bd748f1.json" - }, - { - "match": "or jobs list --folder-key \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\" --output json", - "file": "0a55b81a39.json" - }, - { - "match": "or jobs list --folder-key f1e2d3c4-b5a6-7890-1234-56789abcdef0 --output json", - "file": "0a55b81a39.json" - }, - { - "match": "or jobs get \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\" --output json", - "file": "2be52d8ea7.json" - }, - { - "match": "or jobs get b2c0b2c3-d4e5-4678-9012-3456789bbbbb --output json", - "file": "2be52d8ea7.json" - }, - { - "match": "or jobs history \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\" --output json", - "file": "f27ee3cf71.json" - }, - { - "match": "or jobs history b2c0b2c3-d4e5-4678-9012-3456789bbbbb --output json", - "file": "f27ee3cf71.json" - }, - { - "match": "or jobs logs \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\" --level Error --output json", - "file": "b420de190c.json" - }, - { - "match": "or jobs logs b2c0b2c3-d4e5-4678-9012-3456789bbbbb --level Error --output json", - "file": "b420de190c.json" - }, - { - "match": "or jobs logs \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\" --output json", - "file": "b420de190c.json" - }, - { - "match": "or jobs logs b2c0b2c3-d4e5-4678-9012-3456789bbbbb --output json", - "file": "b420de190c.json" - }, - { - "match": "admin audit tenant events", - "file": "5fb830e5b4.json" - }, - { - "match": "admin audit org events", - "file": "5fb830e5b4.json" - }, - { - "match": "admin audit tenant sources", - "file": "be1ccc7600.json" - }, - { - "match": "admin audit org sources", - "file": "be1ccc7600.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; cache results in _cache/." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { "pattern": "or jobs get b2c0b2c3", "min": 1, "description": "Agent must inspect the failing job to confirm the exit code" }, - { "pattern": "or jobs history b2c0b2c3", "min": 1, "description": "Agent must inspect state history to distinguish operator-initiated Stop from host-side termination" }, - { "pattern": "admin audit", "min": 1, "description": "Agent must query the audit log to identify the actor" } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/uip-fixture.json new file mode 100644 index 0000000000..0bb07b1328 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/data/uip-fixture.json @@ -0,0 +1,166 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--state", + "Faulted", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Manual\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\n \"EndTime\": \"2026-05-19T08:03:01.255Z\",\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobLogs\",\n \"Data\": [\n {\n \"Level\": \"Error\",\n \"TimeStamp\": \"2026-05-19T08:03:01.255Z\",\n \"Message\": \"Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed.\"\n },\n {\n \"Level\": \"Warn\",\n \"TimeStamp\": \"2026-05-19T08:03:00.005Z\",\n \"Message\": \"[Robot] Stop request received for job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (strategy=Kill, source=API). Terminating executor process.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.520Z\",\n \"Message\": \"[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.510Z\",\n \"Message\": \"Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.310Z\",\n \"Message\": \"Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' created. Source=Schedule, ReleaseName=LongRunningProcess, RequiresUserInteraction=False.\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1e2d3c4-b5a6-7890-1234-56789abcdef0", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobList\",\n \"Data\": [\n {\n \"Key\": \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\",\n \"State\": \"Faulted\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\n \"EndTime\": \"2026-05-19T08:03:01.255Z\",\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\"\n },\n {\n \"Key\": \"9aa0b2c3-d4e5-4678-9012-3456789aaaa1\",\n \"State\": \"Successful\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-18T08:00:00.500Z\",\n \"EndTime\": \"2026-05-18T08:05:00.812Z\",\n \"CreationTime\": \"2026-05-18T08:00:00.295Z\"\n },\n {\n \"Key\": \"8aa0b2c3-d4e5-4678-9012-3456789aaaa2\",\n \"State\": \"Successful\",\n \"ProcessName\": \"LongRunningProcess\",\n \"ReleaseKey\": \"\",\n \"Source\": \"Schedule\",\n \"Type\": \"Unattended\",\n \"Priority\": \"Normal\",\n \"Machine\": \"MOCK-HOST\",\n \"MachineKey\": \"\",\n \"StartTime\": \"2026-05-17T08:00:00.500Z\",\n \"EndTime\": \"2026-05-17T08:05:00.755Z\",\n \"CreationTime\": \"2026-05-17T08:00:00.310Z\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\",\r\n \"StartTime\": \"2026-05-19T08:00:00.500Z\",\r\n \"EndTime\": \"2026-05-19T08:03:01.255Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"8e3f7c41-1a2b-4c5d-8e6f-0a1b2c3d4e5f\",\r\n \"Info\": \"System.Exception: Job stopped with an unexpected exit code: 0x40010004\\r\\n at UiPath.Robot.Service.Hosting.ExecutorSupervisor.HandleExitAsync(Int32 exitCode, CancellationToken cancellationToken)\\r\\n at UiPath.Robot.Service.JobManager.FinalizeJobAsync(JobRecord record, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-05-19T08:00:00.302Z\",\r\n \"ReleaseName\": \"LongRunningProcess\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5670099,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\AUTOMATION1\",\r\n \"OrchestratorUserIdentity\": \"automation1\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-19T08:03:01.255Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\",\r\n \"Id\": 670911044\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobHistory\",\n \"Data\": [\n {\n \"State\": \"Pending\",\n \"Timestamp\": \"2026-05-19T08:00:00.302Z\",\n \"Reason\": \"JobCreated\"\n },\n {\n \"State\": \"Running\",\n \"Timestamp\": \"2026-05-19T08:00:00.500Z\",\n \"Reason\": \"ExecutorStarted\"\n },\n {\n \"State\": \"Killing\",\n \"Timestamp\": \"2026-05-19T08:03:00.000Z\",\n \"Reason\": \"StopRequested\",\n \"Strategy\": \"Kill\",\n \"Source\": \"API\"\n },\n {\n \"State\": \"Faulted\",\n \"Timestamp\": \"2026-05-19T08:03:01.255Z\",\n \"Reason\": \"ExecutorExited\",\n \"ExitCode\": \"0x40010004\"\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "b2c0b2c3-d4e5-4678-9012-3456789bbbbb", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"JobLogs\",\n \"Data\": [\n {\n \"Level\": \"Error\",\n \"TimeStamp\": \"2026-05-19T08:03:01.255Z\",\n \"Message\": \"Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (ReleaseName=LongRunningProcess) ended with unexpected exit code 0x40010004. The executor was terminated externally; no graceful shutdown was performed.\"\n },\n {\n \"Level\": \"Warn\",\n \"TimeStamp\": \"2026-05-19T08:03:00.005Z\",\n \"Message\": \"[Robot] Stop request received for job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (strategy=Kill, source=API). Terminating executor process.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.520Z\",\n \"Message\": \"[LongRunningProcess] Starting long-running work — 5 minute delay simulates a slow inference or batch step.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.510Z\",\n \"Message\": \"Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' (ReleaseName=LongRunningProcess) entering Running state on MOCK-HOST.\"\n },\n {\n \"Level\": \"Info\",\n \"TimeStamp\": \"2026-05-19T08:00:00.310Z\",\n \"Message\": \"Job 'b2c0b2c3-d4e5-4678-9012-3456789bbbbb' created. Source=Schedule, ReleaseName=LongRunningProcess, RequiresUserInteraction=False.\"\n }\n ]\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "tenant", + "events" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantEvents\",\n \"Data\": {\n \"auditEvents\": [\n {\n \"id\": \"9e5f1e62-2345-4bcd-9def-2b3c4d5e6f7g\",\n \"createdOn\": \"2026-05-19T08:03:00.105Z\",\n \"organizationId\": \"11111111-2222-3333-4444-555555555555\",\n \"organizationName\": \"Acme Corp\",\n \"tenantId\": \"22222222-3333-4444-5555-666666666666\",\n \"tenantName\": \"Acme-Prod\",\n \"actorId\": \"bbbb2222-cccc-3333-dddd-4444eeee5555\",\n \"actorName\": \"watchdog-svc\",\n \"actorEmail\": \"watchdog-svc@acme.com\",\n \"eventType\": \"Job stopped\",\n \"eventSource\": \"Orchestrator\",\n \"eventTarget\": \"Job\",\n \"eventDetails\": \"{\\\"jobKey\\\":\\\"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\\\",\\\"processName\\\":\\\"LongRunningProcess\\\",\\\"strategy\\\":\\\"Kill\\\",\\\"origin\\\":\\\"API\\\",\\\"folderKey\\\":\\\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\\\",\\\"folderName\\\":\\\"KillScenarios\\\",\\\"actorType\\\":\\\"ServiceAccount\\\",\\\"watchdogRule\\\":\\\"HungJobReaper\\\",\\\"watchdogThresholdSeconds\\\":180,\\\"clientApp\\\":\\\"watchdog-svc/1.4.2\\\"}\",\n \"eventSummary\": \"Job 'LongRunningProcess' (b2c0b2c3-d4e5-4678-9012-3456789bbbbb) was stopped (strategy Kill) by service account watchdog-svc@acme.com via API (rule: HungJobReaper, threshold: 180s)\",\n \"status\": 0\n }\n ],\n \"next\": null,\n \"previous\": null\n }\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "org", + "events" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantEvents\",\n \"Data\": {\n \"auditEvents\": [\n {\n \"id\": \"9e5f1e62-2345-4bcd-9def-2b3c4d5e6f7g\",\n \"createdOn\": \"2026-05-19T08:03:00.105Z\",\n \"organizationId\": \"11111111-2222-3333-4444-555555555555\",\n \"organizationName\": \"Acme Corp\",\n \"tenantId\": \"22222222-3333-4444-5555-666666666666\",\n \"tenantName\": \"Acme-Prod\",\n \"actorId\": \"bbbb2222-cccc-3333-dddd-4444eeee5555\",\n \"actorName\": \"watchdog-svc\",\n \"actorEmail\": \"watchdog-svc@acme.com\",\n \"eventType\": \"Job stopped\",\n \"eventSource\": \"Orchestrator\",\n \"eventTarget\": \"Job\",\n \"eventDetails\": \"{\\\"jobKey\\\":\\\"b2c0b2c3-d4e5-4678-9012-3456789bbbbb\\\",\\\"processName\\\":\\\"LongRunningProcess\\\",\\\"strategy\\\":\\\"Kill\\\",\\\"origin\\\":\\\"API\\\",\\\"folderKey\\\":\\\"f1e2d3c4-b5a6-7890-1234-56789abcdef0\\\",\\\"folderName\\\":\\\"KillScenarios\\\",\\\"actorType\\\":\\\"ServiceAccount\\\",\\\"watchdogRule\\\":\\\"HungJobReaper\\\",\\\"watchdogThresholdSeconds\\\":180,\\\"clientApp\\\":\\\"watchdog-svc/1.4.2\\\"}\",\n \"eventSummary\": \"Job 'LongRunningProcess' (b2c0b2c3-d4e5-4678-9012-3456789bbbbb) was stopped (strategy Kill) by service account watchdog-svc@acme.com via API (rule: HungJobReaper, threshold: 180s)\",\n \"status\": 0\n }\n ],\n \"next\": null,\n \"previous\": null\n }\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "tenant", + "sources" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantSources\",\n \"Data\": [\n {\n \"id\": \"33333333-aaaa-4444-bbbb-555555555555\",\n \"name\": \"Orchestrator\",\n \"targets\": [\n {\n \"id\": \"44444444-cccc-5555-dddd-666666666666\",\n \"name\": \"Job\"\n }\n ],\n \"types\": [\n {\n \"id\": \"55555555-eeee-6666-ffff-777777777777\",\n \"name\": \"Job stopped\"\n }\n ]\n }\n ]\n}\n" + }, + { + "argv": [ + "admin", + "audit", + "org", + "sources" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\n \"Result\": \"Success\",\n \"Code\": \"AuditTenantSources\",\n \"Data\": [\n {\n \"id\": \"33333333-aaaa-4444-bbbb-555555555555\",\n \"name\": \"Orchestrator\",\n \"targets\": [\n {\n \"id\": \"44444444-cccc-5555-dddd-666666666666\",\n \"name\": \"Job\"\n }\n ],\n \"types\": [\n {\n \"id\": \"55555555-eeee-6666-ffff-777777777777\",\n \"name\": \"Job stopped\"\n }\n ]\n }\n ]\n}\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"f1e2d3c4-b5a6-7890-1234-56789abcdef0\",\r\n \"Name\": \"KillScenarios\",\r\n \"Path\": \"KillScenarios\",\r\n \"Description\": \"Folder hosting LongRunningProcess — long-running job used to exercise the 0x40010004 (Job killed) playbook variants.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs get b2c0b2c3", + "min": 1, + "description": "Agent must inspect the failing job to confirm the exit code" + }, + { + "pattern": "or jobs history b2c0b2c3", + "min": 1, + "description": "Agent must inspect state history to distinguish operator-initiated Stop from host-side termination" + }, + { + "pattern": "admin audit", + "min": 1, + "description": "Agent must query the audit log to identify the actor" + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/task.yaml index fe9cc03d0a..d9b610acbe 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-killed-by-watchdog-account/task.yaml @@ -24,21 +24,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/README.md new file mode 100644 index 0000000000..8251b169f4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/README.md @@ -0,0 +1,54 @@ +# Could Not Retrieve Result — Output Too Large + +Reproduces the **output-too-large** case of the `job-output-too-large` +playbook: a job runs to completion but faults at result hand-back +because a bulk `OutputArguments` payload exceeds the result-message +limit. + +``` +Could not retrieve the result of the job execution. This might be +because a message was too large to process. +``` + +## What this scenario uncovers + +**Root Cause:** `InvoiceExtractor` returns a ~184,000-row DataTable +as the output argument `Out_AllInvoices`. The workflow finishes (all +activities Succeeded in traces), but the executor cannot return that +payload to Orchestrator. Fix: write the payload to a Storage Bucket +/ queue / Data Fabric and return only a reference. + +Maps to: +`references/products/orchestrator/playbooks/job-output-too-large.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal UiPath project that declares a `DataTable` output argument `Out_AllInvoices` | + +> Fixtures authored from the playbook signature, not captured from a +> real session. + +## Distinguishing fingerprint + +The Info names *result retrieval / message size*, and `jobs traces` +shows the workflow **completed** — so this is a result hand-back +failure (output too large), not a mid-run logic fault. The specific +bulk output argument (`Out_AllInvoices`, DataTable) is read from the +process source; the graded fix is returning a reference instead of +the payload. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the oversized DataTable output argument as the + root cause (workflow completed; hand-back failed) and recommended + returning a reference (Storage Bucket / queue / Data Fabric) + instead of the bulk payload. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/04a0363cd0.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/04a0363cd0.json deleted file mode 100644 index b50063d97c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/04a0363cd0.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T04:03:52.630Z", - "Message": "Job 'bbdd5566-77ee-88ff-99aa-00bb11223344' faulted. Could not retrieve the result of the job execution. This might be because a message was too large to process." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-24T04:03:52.400Z", - "Message": "[Robot] Failed to serialize and return the job result: output argument 'Out_AllInvoices' (System.Data.DataTable, ~184000 rows)." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-24T04:03:50.900Z", - "Message": "[InvoiceExtractor] Extraction complete; returning DataTable." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/32627accca.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/32627accca.json deleted file mode 100644 index b50063d97c..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/32627accca.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T04:03:52.630Z", - "Message": "Job 'bbdd5566-77ee-88ff-99aa-00bb11223344' faulted. Could not retrieve the result of the job execution. This might be because a message was too large to process." - }, - { - "Level": "Error", - "TimeStamp": "2026-06-24T04:03:52.400Z", - "Message": "[Robot] Failed to serialize and return the job result: output argument 'Out_AllInvoices' (System.Data.DataTable, ~184000 rows)." - }, - { - "Level": "Info", - "TimeStamp": "2026-06-24T04:03:50.900Z", - "Message": "[InvoiceExtractor] Extraction complete; returning DataTable." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/34f4b4e050.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/34f4b4e050.json deleted file mode 100644 index cf44e17409..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/34f4b4e050.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - {"Activity": "Main Sequence", "Status": "Succeeded", "StartTime": "2026-06-24T04:00:02.010Z", "EndTime": "2026-06-24T04:03:50.980Z"}, - {"Activity": "Extract all invoices", "Status": "Succeeded", "StartTime": "2026-06-24T04:00:02.220Z", "EndTime": "2026-06-24T04:03:49.700Z"}, - {"Activity": "Assign Out_AllInvoices", "Status": "Succeeded", "StartTime": "2026-06-24T04:03:49.720Z", "EndTime": "2026-06-24T04:03:50.960Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/3c24c1e838.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/3c24c1e838.json deleted file mode 100644 index cf44e17409..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/3c24c1e838.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - {"Activity": "Main Sequence", "Status": "Succeeded", "StartTime": "2026-06-24T04:00:02.010Z", "EndTime": "2026-06-24T04:03:50.980Z"}, - {"Activity": "Extract all invoices", "Status": "Succeeded", "StartTime": "2026-06-24T04:00:02.220Z", "EndTime": "2026-06-24T04:03:49.700Z"}, - {"Activity": "Assign Out_AllInvoices", "Status": "Succeeded", "StartTime": "2026-06-24T04:03:49.720Z", "EndTime": "2026-06-24T04:03:50.960Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/48f4fc3d2b.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/48f4fc3d2b.json deleted file mode 100644 index 6333fb06ac..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/48f4fc3d2b.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "bbdd5566-77ee-88ff-99aa-00bb11223344", - "State": "Faulted", - "ProcessName": "InvoiceExtractor", - "Source": "Schedule", - "Type": "Unattended", - "RuntimeType": "Unattended", - "Machine": "MOCK-HOST", - "StartTime": "2026-06-24T04:00:01.100Z", - "EndTime": "2026-06-24T04:03:52.640Z", - "CreationTime": "2026-06-24T04:00:01.000Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/8707fcc868.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/8707fcc868.json deleted file mode 100644 index 71c9025868..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/8707fcc868.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bbdd5566-77ee-88ff-99aa-00bb11223344", - "StartTime": "2026-06-24T04:00:01.100Z", - "EndTime": "2026-06-24T04:03:52.640Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Could not retrieve the result of the job execution. This might be because a message was too large to process.", - "CreationTime": "2026-06-24T04:00:01.000Z", - "ReleaseName": "InvoiceExtractor", - "Type": "Unattended", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OutputArguments": "", - "HostMachineName": "MOCK-HOST", - "OrganizationUnitId": 7791630, - "FolderKey": "f4a4b5c6-d7e8-9012-3def-456789012345", - "ProcessType": "Process", - "LastModificationTime": "2026-06-24T04:03:52.640Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "bbdd5566-77ee-88ff-99aa-00bb11223344", - "Id": 671550011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/README.md deleted file mode 100644 index 4f21bc8ded..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Could Not Retrieve Result — Output Too Large - -Reproduces the **output-too-large** case of the `job-output-too-large` -playbook: a job runs to completion but faults at result hand-back -because a bulk `OutputArguments` payload exceeds the result-message -limit. - -``` -Could not retrieve the result of the job execution. This might be -because a message was too large to process. -``` - -## What this scenario uncovers - -**Root Cause:** `InvoiceExtractor` returns a ~184,000-row DataTable -as the output argument `Out_AllInvoices`. The workflow finishes (all -activities Succeeded in traces), but the executor cannot return that -payload to Orchestrator. Fix: write the payload to a Storage Bucket -/ queue / Data Fabric and return only a reference. - -Maps to: -`references/products/orchestrator/playbooks/job-output-too-large.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal UiPath project that declares a `DataTable` output argument `Out_AllInvoices` | -| `data/m/r/*.json` | **synthetic** canned `uip` responses (jobs get/list/logs, **jobs traces** proving the workflow completed) | -| `data/m/r/manifest.json` | dispatch table | - -> Fixtures authored from the playbook signature, not captured from a -> real session. - -## Distinguishing fingerprint - -The Info names *result retrieval / message size*, and `jobs traces` -shows the workflow **completed** — so this is a result hand-back -failure (output too large), not a mid-run logic fault. The specific -bulk output argument (`Out_AllInvoices`, DataTable) is read from the -process source; the graded fix is returning a reference instead of -the payload. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the oversized DataTable output argument as the - root cause (workflow completed; hand-back failed) and recommended - returning a reference (Storage Bucket / queue / Data Fabric) - instead of the bulk payload. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/ee2628e6b6.json deleted file mode 100644 index 1f4fdd39e2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - {"Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", "Name": "original_email@test.com's workspace", "Path": "original_email@test.com's workspace", "Type": "Personal", "ParentKey": ""}, - {"Key": "f4a4b5c6-d7e8-9012-3def-456789012345", "Name": "FinanceOps", "Path": "FinanceOps", "Description": "Folder hosting the InvoiceExtractor process.", "Type": "Standard", "ParentKey": ""}, - {"Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", "Name": "Shared", "Path": "Shared", "Type": "Standard", "ParentKey": ""} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/manifest.json deleted file mode 100644 index 3b94e4b2c5..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/m/r/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the output-too-large scenario (job-output-too-large playbook). The agent chains: folders list -> jobs list --state Faulted -> jobs get (Info: could not retrieve result / message too large; OutputArguments empty) -> jobs logs (workflow completed, then handback failed on Out_AllInvoices DataTable) -> jobs traces (all activities Succeeded) and reads process/project.json + entry-points.json for the DataTable output argument. Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key f4a4b5c6-d7e8-9012-3def-456789012345 --state Faulted", "file": "48f4fc3d2b.json" }, - { "match": "or jobs get bbdd5566-77ee-88ff-99aa-00bb11223344", "file": "8707fcc868.json" }, - { "match": "or jobs logs bbdd5566-77ee-88ff-99aa-00bb11223344 --level Error", "file": "32627accca.json" }, - { "match": "or jobs logs bbdd5566-77ee-88ff-99aa-00bb11223344", "file": "04a0363cd0.json" }, - { "match": "or jobs traces bbdd5566-77ee-88ff-99aa-00bb11223344", "file": "3c24c1e838.json" }, - { "match": "traces spans get --job-key bbdd5566-77ee-88ff-99aa-00bb11223344", "file": "34f4b4e050.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0 } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/uip-fixture.json new file mode 100644 index 0000000000..2323d38afe --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/data/uip-fixture.json @@ -0,0 +1,91 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f4a4b5c6-d7e8-9012-3def-456789012345", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"bbdd5566-77ee-88ff-99aa-00bb11223344\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceExtractor\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-06-24T04:00:01.100Z\",\r\n \"EndTime\": \"2026-06-24T04:03:52.640Z\",\r\n \"CreationTime\": \"2026-06-24T04:00:01.000Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bbdd5566-77ee-88ff-99aa-00bb11223344", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T04:03:52.630Z\",\r\n \"Message\": \"Job 'bbdd5566-77ee-88ff-99aa-00bb11223344' faulted. Could not retrieve the result of the job execution. This might be because a message was too large to process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T04:03:52.400Z\",\r\n \"Message\": \"[Robot] Failed to serialize and return the job result: output argument 'Out_AllInvoices' (System.Data.DataTable, ~184000 rows).\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-24T04:03:50.900Z\",\r\n \"Message\": \"[InvoiceExtractor] Extraction complete; returning DataTable.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "bbdd5566-77ee-88ff-99aa-00bb11223344" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\"Activity\": \"Main Sequence\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-24T04:00:02.010Z\", \"EndTime\": \"2026-06-24T04:03:50.980Z\"},\r\n {\"Activity\": \"Extract all invoices\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-24T04:00:02.220Z\", \"EndTime\": \"2026-06-24T04:03:49.700Z\"},\r\n {\"Activity\": \"Assign Out_AllInvoices\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-24T04:03:49.720Z\", \"EndTime\": \"2026-06-24T04:03:50.960Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "bbdd5566-77ee-88ff-99aa-00bb11223344" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bbdd5566-77ee-88ff-99aa-00bb11223344\",\r\n \"StartTime\": \"2026-06-24T04:00:01.100Z\",\r\n \"EndTime\": \"2026-06-24T04:03:52.640Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Could not retrieve the result of the job execution. This might be because a message was too large to process.\",\r\n \"CreationTime\": \"2026-06-24T04:00:01.000Z\",\r\n \"ReleaseName\": \"InvoiceExtractor\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OutputArguments\": \"\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"OrganizationUnitId\": 7791630,\r\n \"FolderKey\": \"f4a4b5c6-d7e8-9012-3def-456789012345\",\r\n \"ProcessType\": \"Process\",\r\n \"LastModificationTime\": \"2026-06-24T04:03:52.640Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bbdd5566-77ee-88ff-99aa-00bb11223344\",\r\n \"Id\": 671550011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bbdd5566-77ee-88ff-99aa-00bb11223344" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T04:03:52.630Z\",\r\n \"Message\": \"Job 'bbdd5566-77ee-88ff-99aa-00bb11223344' faulted. Could not retrieve the result of the job execution. This might be because a message was too large to process.\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T04:03:52.400Z\",\r\n \"Message\": \"[Robot] Failed to serialize and return the job result: output argument 'Out_AllInvoices' (System.Data.DataTable, ~184000 rows).\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-06-24T04:03:50.900Z\",\r\n \"Message\": \"[InvoiceExtractor] Extraction complete; returning DataTable.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "bbdd5566-77ee-88ff-99aa-00bb11223344" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\"Activity\": \"Main Sequence\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-24T04:00:02.010Z\", \"EndTime\": \"2026-06-24T04:03:50.980Z\"},\r\n {\"Activity\": \"Extract all invoices\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-24T04:00:02.220Z\", \"EndTime\": \"2026-06-24T04:03:49.700Z\"},\r\n {\"Activity\": \"Assign Out_AllInvoices\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-24T04:03:49.720Z\", \"EndTime\": \"2026-06-24T04:03:50.960Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\", \"Name\": \"original_email@test.com's workspace\", \"Path\": \"original_email@test.com's workspace\", \"Type\": \"Personal\", \"ParentKey\": \"\"},\r\n {\"Key\": \"f4a4b5c6-d7e8-9012-3def-456789012345\", \"Name\": \"FinanceOps\", \"Path\": \"FinanceOps\", \"Description\": \"Folder hosting the InvoiceExtractor process.\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\", \"Name\": \"Shared\", \"Path\": \"Shared\", \"Type\": \"Standard\", \"ParentKey\": \"\"}\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/task.yaml index be4865c782..b191d21a1a 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-output-too-large/task.yaml @@ -26,21 +26,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/README.md new file mode 100644 index 0000000000..0d6951270b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/README.md @@ -0,0 +1,61 @@ +# Job Stopped — Exit Code 0xE0434352 (hidden KeyNotFoundException) + +Reproduces the `0xE0434352` branch of the **Job Stopped — +Unexpected Exit Code (generic)** playbook: an unhandled managed .NET +exception escapes to the process boundary and collapses to the raw +CLR exit code. The job shows only: + +``` +System.Exception: Job stopped with an unexpected exit code: 0xE0434352 +``` + +The real cause is recoverable only from execution traces. + +## What this scenario uncovers + +**Root Cause:** An unhandled +`System.Collections.Generic.KeyNotFoundException` ("the given key +'EMEA-NORTH' was not present in the dictionary") at the "Lookup +region total" Assign. With no top-level Try/Catch or Global +Exception Handler, it reached the process boundary — so the job Info +and error logs carry only the raw OS/CLR code `0xE0434352`, and the +real error lives in `uip or jobs traces`. + +Maps to: +`references/products/orchestrator/playbooks/job-stopped-generic-exit-code.md` +(0xE0434352 branch). Also exercises the `KeyNotFoundException` +"given key was not present in the dictionary" signature. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal unattended UiPath project (LogMessage + Delay) | + +> Fixtures are authored from the playbook signature, not captured +> from a real session. Regenerate via +> `_shared/scripts/generate_scenario.py` before treating the score +> as a strict regression signal. + +## Distinguishing fingerprints + +| Code / playbook | Fingerprint that rules it out here | +|---|---| +| `0x40010004` external kill | Different code; that playbook is for `TerminateProcess`. Here the process died from its own managed exception. | +| `0xC0000005` native access violation | The traced exception is managed (`KeyNotFoundException`), not a native crash. | +| `0xC000026B` session teardown | No logoff/shutdown; fault is in workflow logic. | +| **`0xE0434352`** *(this scenario)* | Traces show an unhandled managed exception (`KeyNotFoundException`) that escaped to the process boundary. | + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent did NOT stop at the raw code or merely suggest a rerun — it + recovered the real `KeyNotFoundException` (missing dictionary key) + from traces, and recommended guarding the lookup and adding + top-level exception handling (not blaming the robot version). + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/0c37d6b759.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/0c37d6b759.json deleted file mode 100644 index c953282116..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/0c37d6b759.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-22T06:00:39.860Z", - "Message": "Job 'aacc44ee-55ff-66aa-77bb-88cc99001122' faulted. System.Exception: Job stopped with an unexpected exit code: 0xE0434352" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-22T06:00:39.700Z", - "Message": "[Robot] Executor process exited with code 0xE0434352 (CLR unhandled managed exception). Inspect execution traces for the managed exception detail." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/11d49febab.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/11d49febab.json deleted file mode 100644 index c953282116..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/11d49febab.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-22T06:00:39.860Z", - "Message": "Job 'aacc44ee-55ff-66aa-77bb-88cc99001122' faulted. System.Exception: Job stopped with an unexpected exit code: 0xE0434352" - }, - { - "Level": "Error", - "TimeStamp": "2026-06-22T06:00:39.700Z", - "Message": "[Robot] Executor process exited with code 0xE0434352 (CLR unhandled managed exception). Inspect execution traces for the managed exception detail." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/43fefe681e.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/43fefe681e.json deleted file mode 100644 index 57587ebb6a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/43fefe681e.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "Activity": "Main Sequence", - "Status": "Faulted", - "StartTime": "2026-06-22T06:00:03.010Z", - "EndTime": "2026-06-22T06:00:39.720Z" - }, - { - "Activity": "Read region config", - "Status": "Succeeded", - "StartTime": "2026-06-22T06:00:03.220Z", - "EndTime": "2026-06-22T06:00:12.540Z" - }, - { - "Activity": "Lookup region total (Assign)", - "Status": "Faulted", - "StartTime": "2026-06-22T06:00:39.610Z", - "EndTime": "2026-06-22T06:00:39.700Z", - "Error": { - "Type": "System.Collections.Generic.KeyNotFoundException", - "Message": "The given key 'EMEA-NORTH' was not present in the dictionary.", - "StackTrace": "at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n at lambda_method(Closure, ActivityContext)\r\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)", - "Handled": false - } - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/50bc27043c.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/50bc27043c.json deleted file mode 100644 index 84bc1c17e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/50bc27043c.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "aacc44ee-55ff-66aa-77bb-88cc99001122", - "State": "Faulted", - "ProcessName": "ReportCompiler", - "Source": "Schedule", - "Type": "Unattended", - "RuntimeType": "Unattended", - "Machine": "MOCK-HOST", - "StartTime": "2026-06-22T06:00:02.500Z", - "EndTime": "2026-06-22T06:00:39.870Z", - "CreationTime": "2026-06-22T06:00:02.300Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/76a86bdeea.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/76a86bdeea.json deleted file mode 100644 index 57587ebb6a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/76a86bdeea.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "Activity": "Main Sequence", - "Status": "Faulted", - "StartTime": "2026-06-22T06:00:03.010Z", - "EndTime": "2026-06-22T06:00:39.720Z" - }, - { - "Activity": "Read region config", - "Status": "Succeeded", - "StartTime": "2026-06-22T06:00:03.220Z", - "EndTime": "2026-06-22T06:00:12.540Z" - }, - { - "Activity": "Lookup region total (Assign)", - "Status": "Faulted", - "StartTime": "2026-06-22T06:00:39.610Z", - "EndTime": "2026-06-22T06:00:39.700Z", - "Error": { - "Type": "System.Collections.Generic.KeyNotFoundException", - "Message": "The given key 'EMEA-NORTH' was not present in the dictionary.", - "StackTrace": "at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n at lambda_method(Closure, ActivityContext)\r\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)", - "Handled": false - } - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/7a89e31cab.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/7a89e31cab.json deleted file mode 100644 index 997ea45bb2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/7a89e31cab.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "aacc44ee-55ff-66aa-77bb-88cc99001122", - "StartTime": "2026-06-22T06:00:02.500Z", - "EndTime": "2026-06-22T06:00:39.870Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "System.Exception: Job stopped with an unexpected exit code: 0xE0434352", - "CreationTime": "2026-06-22T06:00:02.300Z", - "ReleaseName": "ReportCompiler", - "Type": "Unattended", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "OrganizationUnitId": 7791625, - "FolderKey": "e3c3d4e5-f6a7-8901-2cde-f01234567890", - "ProcessType": "Process", - "MachineName": "MOCK-HOST", - "MachineKey": "f0e1d2c3-b4a5-6978-8c9d-0e1f2a3b4c5d", - "LastModificationTime": "2026-06-22T06:00:39.870Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "aacc44ee-55ff-66aa-77bb-88cc99001122", - "Id": 671440011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/README.md deleted file mode 100644 index 23c12d5b2e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/README.md +++ /dev/null @@ -1,60 +0,0 @@ -# Job Stopped — Exit Code 0xE0434352 (hidden KeyNotFoundException) - -Reproduces the `0xE0434352` branch of the **Job Stopped — -Unexpected Exit Code (generic)** playbook: an unhandled managed .NET -exception escapes to the process boundary and collapses to the raw -CLR exit code. The job shows only: - -``` -System.Exception: Job stopped with an unexpected exit code: 0xE0434352 -``` - -The real cause is recoverable only from execution traces. - -## What this scenario uncovers - -**Root Cause:** An unhandled -`System.Collections.Generic.KeyNotFoundException` ("the given key -'EMEA-NORTH' was not present in the dictionary") at the "Lookup -region total" Assign. With no top-level Try/Catch or Global -Exception Handler, it reached the process boundary — so the job Info -and error logs carry only the raw OS/CLR code `0xE0434352`, and the -real error lives in `uip or jobs traces`. - -Maps to: -`references/products/orchestrator/playbooks/job-stopped-generic-exit-code.md` -(0xE0434352 branch). Also exercises the `KeyNotFoundException` -"given key was not present in the dictionary" signature. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal unattended UiPath project (LogMessage + Delay) | -| `data/m/r/*.json` | **synthetic** canned `uip` responses (jobs get/list/logs, **jobs traces** carrying the real exception, machines list) | -| `data/m/r/manifest.json` | dispatch table mapping each command to its fixture | - -> Fixtures are authored from the playbook signature, not captured -> from a real session. Regenerate via -> `_shared/scripts/generate_scenario.py` before treating the score -> as a strict regression signal. - -## Distinguishing fingerprints - -| Code / playbook | Fingerprint that rules it out here | -|---|---| -| `0x40010004` external kill | Different code; that playbook is for `TerminateProcess`. Here the process died from its own managed exception. | -| `0xC0000005` native access violation | The traced exception is managed (`KeyNotFoundException`), not a native crash. | -| `0xC000026B` session teardown | No logoff/shutdown; fault is in workflow logic. | -| **`0xE0434352`** *(this scenario)* | Traces show an unhandled managed exception (`KeyNotFoundException`) that escaped to the process boundary. | - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent did NOT stop at the raw code or merely suggest a rerun — it - recovered the real `KeyNotFoundException` (missing dictionary key) - from traces, and recommended guarding the lookup and adding - top-level exception handling (not blaming the robot version). diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/dc064823f4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/dc064823f4.json deleted file mode 100644 index e6979fa4b4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/dc064823f4.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "Key": "f0e1d2c3-b4a5-6978-8c9d-0e1f2a3b4c5d", - "Name": "ReportingRuntime", - "Type": "Template", - "Runtimes": 2, - "Hosts": ["MOCK-HOST"], - "RobotVersion": "23.4.0", - "Description": "Runtime template for the ReportingOps folder; hosts run an older robot build." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/ee2628e6b6.json deleted file mode 100644 index bff50197d8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "e3c3d4e5-f6a7-8901-2cde-f01234567890", - "Name": "ReportingOps", - "Path": "ReportingOps", - "Description": "Folder hosting the ReportCompiler unattended process.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/manifest.json deleted file mode 100644 index 7d0ca10a23..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/m/r/manifest.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the generic-exit-code scenario (job-stopped-generic-exit-code playbook). The job Info shows only the raw CLR code 0xE0434352; the REAL cause (an unhandled System.Collections.Generic.KeyNotFoundException) is recovered from job traces. The agent chains: folders list -> jobs list --state Faulted -> jobs get (Info: exit code 0xE0434352) -> jobs logs (raw code, no top-level handler) -> jobs traces (KeyNotFoundException at 'Lookup region total') -> machines list (robot 23.4.0, checked but not the cause). Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key e3c3d4e5-f6a7-8901-2cde-f01234567890 --state Faulted", "file": "50bc27043c.json" }, - { "match": "or jobs get aacc44ee-55ff-66aa-77bb-88cc99001122", "file": "7a89e31cab.json" }, - { "match": "or jobs logs aacc44ee-55ff-66aa-77bb-88cc99001122 --level Error", "file": "0c37d6b759.json" }, - { "match": "or jobs logs aacc44ee-55ff-66aa-77bb-88cc99001122", "file": "11d49febab.json" }, - { "match": "or jobs traces aacc44ee-55ff-66aa-77bb-88cc99001122", "file": "43fefe681e.json" }, - { "match": "traces spans get --job-key aacc44ee-55ff-66aa-77bb-88cc99001122", "file": "76a86bdeea.json" }, - { "match": "or machines list", "file": "dc064823f4.json" }, - { "match": "docsai ask", "passthrough": true, "_doc": "Proxy any docsai query to the real uip CLI; cache in _cache/." } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/uip-fixture.json new file mode 100644 index 0000000000..b666a4d5c8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/data/uip-fixture.json @@ -0,0 +1,101 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "e3c3d4e5-f6a7-8901-2cde-f01234567890", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"aacc44ee-55ff-66aa-77bb-88cc99001122\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"ReportCompiler\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-06-22T06:00:02.500Z\",\r\n \"EndTime\": \"2026-06-22T06:00:39.870Z\",\r\n \"CreationTime\": \"2026-06-22T06:00:02.300Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aacc44ee-55ff-66aa-77bb-88cc99001122", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T06:00:39.860Z\",\r\n \"Message\": \"Job 'aacc44ee-55ff-66aa-77bb-88cc99001122' faulted. System.Exception: Job stopped with an unexpected exit code: 0xE0434352\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T06:00:39.700Z\",\r\n \"Message\": \"[Robot] Executor process exited with code 0xE0434352 (CLR unhandled managed exception). Inspect execution traces for the managed exception detail.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "aacc44ee-55ff-66aa-77bb-88cc99001122" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"Activity\": \"Main Sequence\",\r\n \"Status\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T06:00:03.010Z\",\r\n \"EndTime\": \"2026-06-22T06:00:39.720Z\"\r\n },\r\n {\r\n \"Activity\": \"Read region config\",\r\n \"Status\": \"Succeeded\",\r\n \"StartTime\": \"2026-06-22T06:00:03.220Z\",\r\n \"EndTime\": \"2026-06-22T06:00:12.540Z\"\r\n },\r\n {\r\n \"Activity\": \"Lookup region total (Assign)\",\r\n \"Status\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T06:00:39.610Z\",\r\n \"EndTime\": \"2026-06-22T06:00:39.700Z\",\r\n \"Error\": {\r\n \"Type\": \"System.Collections.Generic.KeyNotFoundException\",\r\n \"Message\": \"The given key 'EMEA-NORTH' was not present in the dictionary.\",\r\n \"StackTrace\": \"at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\\r\\n at lambda_method(Closure, ActivityContext)\\r\\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\",\r\n \"Handled\": false\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "aacc44ee-55ff-66aa-77bb-88cc99001122" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"aacc44ee-55ff-66aa-77bb-88cc99001122\",\r\n \"StartTime\": \"2026-06-22T06:00:02.500Z\",\r\n \"EndTime\": \"2026-06-22T06:00:39.870Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"System.Exception: Job stopped with an unexpected exit code: 0xE0434352\",\r\n \"CreationTime\": \"2026-06-22T06:00:02.300Z\",\r\n \"ReleaseName\": \"ReportCompiler\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"OrganizationUnitId\": 7791625,\r\n \"FolderKey\": \"e3c3d4e5-f6a7-8901-2cde-f01234567890\",\r\n \"ProcessType\": \"Process\",\r\n \"MachineName\": \"MOCK-HOST\",\r\n \"MachineKey\": \"f0e1d2c3-b4a5-6978-8c9d-0e1f2a3b4c5d\",\r\n \"LastModificationTime\": \"2026-06-22T06:00:39.870Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"aacc44ee-55ff-66aa-77bb-88cc99001122\",\r\n \"Id\": 671440011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "aacc44ee-55ff-66aa-77bb-88cc99001122" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T06:00:39.860Z\",\r\n \"Message\": \"Job 'aacc44ee-55ff-66aa-77bb-88cc99001122' faulted. System.Exception: Job stopped with an unexpected exit code: 0xE0434352\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T06:00:39.700Z\",\r\n \"Message\": \"[Robot] Executor process exited with code 0xE0434352 (CLR unhandled managed exception). Inspect execution traces for the managed exception detail.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "aacc44ee-55ff-66aa-77bb-88cc99001122" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"Activity\": \"Main Sequence\",\r\n \"Status\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T06:00:03.010Z\",\r\n \"EndTime\": \"2026-06-22T06:00:39.720Z\"\r\n },\r\n {\r\n \"Activity\": \"Read region config\",\r\n \"Status\": \"Succeeded\",\r\n \"StartTime\": \"2026-06-22T06:00:03.220Z\",\r\n \"EndTime\": \"2026-06-22T06:00:12.540Z\"\r\n },\r\n {\r\n \"Activity\": \"Lookup region total (Assign)\",\r\n \"Status\": \"Faulted\",\r\n \"StartTime\": \"2026-06-22T06:00:39.610Z\",\r\n \"EndTime\": \"2026-06-22T06:00:39.700Z\",\r\n \"Error\": {\r\n \"Type\": \"System.Collections.Generic.KeyNotFoundException\",\r\n \"Message\": \"The given key 'EMEA-NORTH' was not present in the dictionary.\",\r\n \"StackTrace\": \"at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\\r\\n at lambda_method(Closure, ActivityContext)\\r\\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\",\r\n \"Handled\": false\r\n }\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e3c3d4e5-f6a7-8901-2cde-f01234567890\",\r\n \"Name\": \"ReportingOps\",\r\n \"Path\": \"ReportingOps\",\r\n \"Description\": \"Folder hosting the ReportCompiler unattended process.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f0e1d2c3-b4a5-6978-8c9d-0e1f2a3b4c5d\",\r\n \"Name\": \"ReportingRuntime\",\r\n \"Type\": \"Template\",\r\n \"Runtimes\": 2,\r\n \"Hosts\": [\"MOCK-HOST\"],\r\n \"RobotVersion\": \"23.4.0\",\r\n \"Description\": \"Runtime template for the ReportingOps folder; hosts run an older robot build.\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/task.yaml index d035dd38b4..55874ac071 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-job-stopped-exit-code-e0434352/task.yaml @@ -29,21 +29,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/README.md new file mode 100644 index 0000000000..e3047821df --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/README.md @@ -0,0 +1,56 @@ +# Screen Capture Failed — Win32 "The handle is invalid" + +Reproduces the `screen-capture-handle-invalid` playbook: a screenshot +activity throws a Win32 exception because the unattended run has no +live interactive desktop to capture. + +``` +System.ComponentModel.Win32Exception (6): The handle is invalid. + at System.Drawing.Graphics.CopyFromScreen(...) +``` + +## What this scenario uncovers + +**Root Cause:** `DashboardSnapshotBot` runs unattended with +`LoginToConsole: true`, so the robot attaches to the machine's single +physical console session instead of opening its own RDP session; with +no interactive logon at that console there is no desktop for +`CopyFromScreen`. It works when a human is logged in at the console (a +real desktop exists) but fails as a scheduled unattended job. Fix: set +Login to Console to No so the robot opens its own RDP session (or keep +an RDP session connected) with a live desktop. + +Maps to: +`references/products/orchestrator/playbooks/screen-capture-handle-invalid.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal unattended UiPath project (dashboard snapshot) | + +> Fixtures authored from the playbook signature, not captured from a +> real session. + +## Distinguishing fingerprint + +The "works when logged in at the console, fails unattended" clue plus +`LoginToConsole: true` and the `CopyFromScreen` failure point at a +missing interactive desktop surface, not a workflow-logic bug. The +graded fix is giving the robot its own live session (set Login to +Console to No), not editing the workflow. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the missing interactive desktop session + (LoginToConsole on → attached to the physical console, unattended) + as the reason screen capture fails, and recommended setting Login + to Console to No so the robot opens its own RDP session (or keeping + an RDP session connected) rather than changing the workflow logic. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/2c1eba59a4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/2c1eba59a4.json deleted file mode 100644 index f54ac6f2eb..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/2c1eba59a4.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "UserGet", - "Data": { - "Key": "bb22cc33-dd44-ee55-ff66-778899aabbcc", - "UserName": "svcmonitor", - "DomainName": "UIPATH", - "FullName": "Service Account Monitoring", - "Type": "User", - "UnattendedRobot": { - "ExecutionSettings": {"LoginToConsole": true, "RunOnlyOneJobAtATime": false, "ResolutionWidth": 1920, "ResolutionHeight": 1080, "ResolutionDepth": 32}, - "UserName": "UIPATH\\SVCMON", - "CredentialStoreId": null, - "CredentialType": "Default", - "PasswordLastSet": "2026-06-01T09:00:00Z" - }, - "TenantId": 1, - "Id": 7510 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/7d3d027c8a.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/7d3d027c8a.json deleted file mode 100644 index 125cfb95ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/7d3d027c8a.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-29T03:30:09.200Z", "Message": "Take Screenshot: System.ComponentModel.Win32Exception (6): The handle is invalid. at System.Drawing.Graphics.CopyFromScreen(...)"}, - {"Level": "Error", "TimeStamp": "2026-06-29T03:30:09.050Z", "Message": "[Robot] Screen capture requested but no active interactive desktop session is available for this unattended run. CopyFromScreen has no valid device context."}, - {"Level": "Info", "TimeStamp": "2026-06-29T03:30:04.100Z", "Message": "[DashboardSnapshotBot] Capturing dashboard snapshot."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/88b1fb5ee1.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/88b1fb5ee1.json deleted file mode 100644 index 582f052134..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/88b1fb5ee1.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - {"Activity": "Main Sequence", "Status": "Faulted", "StartTime": "2026-06-29T03:30:02.010Z", "EndTime": "2026-06-29T03:30:09.210Z"}, - {"Activity": "Open dashboard", "Status": "Succeeded", "StartTime": "2026-06-29T03:30:02.220Z", "EndTime": "2026-06-29T03:30:08.700Z"}, - {"Activity": "Take Screenshot", "Status": "Faulted", "StartTime": "2026-06-29T03:30:08.720Z", "EndTime": "2026-06-29T03:30:09.200Z", "Error": {"Type": "System.ComponentModel.Win32Exception", "Message": "The handle is invalid.", "Activity": "Take Screenshot", "StackTrace": "at System.Drawing.Graphics.CopyFromScreen(...)"}} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/8e03f98302.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/8e03f98302.json deleted file mode 100644 index 582f052134..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/8e03f98302.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - {"Activity": "Main Sequence", "Status": "Faulted", "StartTime": "2026-06-29T03:30:02.010Z", "EndTime": "2026-06-29T03:30:09.210Z"}, - {"Activity": "Open dashboard", "Status": "Succeeded", "StartTime": "2026-06-29T03:30:02.220Z", "EndTime": "2026-06-29T03:30:08.700Z"}, - {"Activity": "Take Screenshot", "Status": "Faulted", "StartTime": "2026-06-29T03:30:08.720Z", "EndTime": "2026-06-29T03:30:09.200Z", "Error": {"Type": "System.ComponentModel.Win32Exception", "Message": "The handle is invalid.", "Activity": "Take Screenshot", "StackTrace": "at System.Drawing.Graphics.CopyFromScreen(...)"}} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/9a58c51891.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/9a58c51891.json deleted file mode 100644 index cf03635be1..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/9a58c51891.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - {"Key": "ccddeeff-3344-5566-7788-99aabbccddee", "State": "Faulted", "ProcessName": "DashboardSnapshotBot", "Source": "Schedule", "Type": "Unattended", "RuntimeType": "Unattended", "Machine": "MOCK-HOST", "StartTime": "2026-06-29T03:30:01.500Z", "EndTime": "2026-06-29T03:30:09.220Z", "CreationTime": "2026-06-29T03:30:01.400Z"} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/README.md deleted file mode 100644 index 90f697d2e8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/README.md +++ /dev/null @@ -1,55 +0,0 @@ -# Screen Capture Failed — Win32 "The handle is invalid" - -Reproduces the `screen-capture-handle-invalid` playbook: a screenshot -activity throws a Win32 exception because the unattended run has no -live interactive desktop to capture. - -``` -System.ComponentModel.Win32Exception (6): The handle is invalid. - at System.Drawing.Graphics.CopyFromScreen(...) -``` - -## What this scenario uncovers - -**Root Cause:** `DashboardSnapshotBot` runs unattended with -`LoginToConsole: true`, so the robot attaches to the machine's single -physical console session instead of opening its own RDP session; with -no interactive logon at that console there is no desktop for -`CopyFromScreen`. It works when a human is logged in at the console (a -real desktop exists) but fails as a scheduled unattended job. Fix: set -Login to Console to No so the robot opens its own RDP session (or keep -an RDP session connected) with a live desktop. - -Maps to: -`references/products/orchestrator/playbooks/screen-capture-handle-invalid.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` | -| `process/` | minimal unattended UiPath project (dashboard snapshot) | -| `data/m/r/*.json` | **synthetic** canned `uip` responses — jobs get/list/logs, `jobs traces` (Take Screenshot faulted, prior succeeded), `users get` (LoginToConsole=true) | -| `data/m/r/manifest.json` | dispatch table | - -> Fixtures authored from the playbook signature, not captured from a -> real session. - -## Distinguishing fingerprint - -The "works when logged in at the console, fails unattended" clue plus -`LoginToConsole: true` and the `CopyFromScreen` failure point at a -missing interactive desktop surface, not a workflow-logic bug. The -graded fix is giving the robot its own live session (set Login to -Console to No), not editing the workflow. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the missing interactive desktop session - (LoginToConsole on → attached to the physical console, unattended) - as the reason screen capture fails, and recommended setting Login - to Console to No so the robot opens its own RDP session (or keeping - an RDP session connected) rather than changing the workflow logic. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/c4e34690df.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/c4e34690df.json deleted file mode 100644 index fe4a139cce..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/c4e34690df.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "ccddeeff-3344-5566-7788-99aabbccddee", - "StartTime": "2026-06-29T03:30:01.500Z", - "EndTime": "2026-06-29T03:30:09.220Z", - "State": "Faulted", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "System.ComponentModel.Win32Exception (6): The handle is invalid.\r\n at System.Drawing.Graphics.CopyFromScreen(Int32 sourceX, Int32 sourceY, Int32 destinationX, Int32 destinationY, Size blockRegionSize, CopyPixelOperation copyPixelOperation)", - "CreationTime": "2026-06-29T03:30:01.400Z", - "ReleaseName": "DashboardSnapshotBot", - "Type": "Unattended", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "OrganizationUnitId": 7791650, - "FolderKey": "a7e7f8a9-b0c1-2345-6789-abcdef012345", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\SVCMON", - "OrchestratorUserIdentity": "svcmonitor", - "LastModificationTime": "2026-06-29T03:30:09.220Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "ccddeeff-3344-5566-7788-99aabbccddee", - "Id": 672000011 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/ee2628e6b6.json deleted file mode 100644 index 47a66f27e9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - {"Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", "Name": "original_email@test.com's workspace", "Path": "original_email@test.com's workspace", "Type": "Personal", "ParentKey": ""}, - {"Key": "a7e7f8a9-b0c1-2345-6789-abcdef012345", "Name": "MonitoringOps", "Path": "MonitoringOps", "Description": "Folder hosting the DashboardSnapshotBot unattended process.", "Type": "Standard", "ParentKey": ""}, - {"Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", "Name": "Shared", "Path": "Shared", "Type": "Standard", "ParentKey": ""} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/f301adfb55.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/f301adfb55.json deleted file mode 100644 index 125cfb95ec..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/f301adfb55.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - {"Level": "Error", "TimeStamp": "2026-06-29T03:30:09.200Z", "Message": "Take Screenshot: System.ComponentModel.Win32Exception (6): The handle is invalid. at System.Drawing.Graphics.CopyFromScreen(...)"}, - {"Level": "Error", "TimeStamp": "2026-06-29T03:30:09.050Z", "Message": "[Robot] Screen capture requested but no active interactive desktop session is available for this unattended run. CopyFromScreen has no valid device context."}, - {"Level": "Info", "TimeStamp": "2026-06-29T03:30:04.100Z", "Message": "[DashboardSnapshotBot] Capturing dashboard snapshot."} - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/manifest.json deleted file mode 100644 index eba1bc014a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/m/r/manifest.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the screen-capture handle-invalid scenario (screen-capture-handle-invalid playbook). The agent chains: folders list -> jobs list --state Faulted -> jobs get (Info: Win32Exception handle invalid at CopyFromScreen) -> jobs logs (no active interactive desktop) -> jobs traces (Take Screenshot faulted, prior activities Succeeded) -> users get (LoginToConsole=true, attached to the physical console, unattended). Fix: set Login to Console = false so the robot uses its own RDP session / ensure a live connected session. Keys are stable.", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key a7e7f8a9-b0c1-2345-6789-abcdef012345 --state Faulted", "file": "9a58c51891.json" }, - { "match": "or jobs get ccddeeff-3344-5566-7788-99aabbccddee", "file": "c4e34690df.json" }, - { "match": "or jobs logs ccddeeff-3344-5566-7788-99aabbccddee --level Error", "file": "7d3d027c8a.json" }, - { "match": "or jobs logs ccddeeff-3344-5566-7788-99aabbccddee", "file": "f301adfb55.json" }, - { "match": "or jobs traces ccddeeff-3344-5566-7788-99aabbccddee", "file": "88b1fb5ee1.json" }, - { "match": "traces spans get --job-key ccddeeff-3344-5566-7788-99aabbccddee", "file": "8e03f98302.json" }, - { "match": "or users get bb22cc33-dd44-ee55-ff66-778899aabbcc", "file": "2c1eba59a4.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { "response": "[]\n", "exit_code": 0 } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/uip-fixture.json new file mode 100644 index 0000000000..4889f248a5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/data/uip-fixture.json @@ -0,0 +1,102 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a7e7f8a9-b0c1-2345-6789-abcdef012345", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\"Key\": \"ccddeeff-3344-5566-7788-99aabbccddee\", \"State\": \"Faulted\", \"ProcessName\": \"DashboardSnapshotBot\", \"Source\": \"Schedule\", \"Type\": \"Unattended\", \"RuntimeType\": \"Unattended\", \"Machine\": \"MOCK-HOST\", \"StartTime\": \"2026-06-29T03:30:01.500Z\", \"EndTime\": \"2026-06-29T03:30:09.220Z\", \"CreationTime\": \"2026-06-29T03:30:01.400Z\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ccddeeff-3344-5566-7788-99aabbccddee", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-29T03:30:09.200Z\", \"Message\": \"Take Screenshot: System.ComponentModel.Win32Exception (6): The handle is invalid. at System.Drawing.Graphics.CopyFromScreen(...)\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-29T03:30:09.050Z\", \"Message\": \"[Robot] Screen capture requested but no active interactive desktop session is available for this unattended run. CopyFromScreen has no valid device context.\"},\r\n {\"Level\": \"Info\", \"TimeStamp\": \"2026-06-29T03:30:04.100Z\", \"Message\": \"[DashboardSnapshotBot] Capturing dashboard snapshot.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "traces", + "spans", + "get", + "--job-key", + "ccddeeff-3344-5566-7788-99aabbccddee" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\"Activity\": \"Main Sequence\", \"Status\": \"Faulted\", \"StartTime\": \"2026-06-29T03:30:02.010Z\", \"EndTime\": \"2026-06-29T03:30:09.210Z\"},\r\n {\"Activity\": \"Open dashboard\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-29T03:30:02.220Z\", \"EndTime\": \"2026-06-29T03:30:08.700Z\"},\r\n {\"Activity\": \"Take Screenshot\", \"Status\": \"Faulted\", \"StartTime\": \"2026-06-29T03:30:08.720Z\", \"EndTime\": \"2026-06-29T03:30:09.200Z\", \"Error\": {\"Type\": \"System.ComponentModel.Win32Exception\", \"Message\": \"The handle is invalid.\", \"Activity\": \"Take Screenshot\", \"StackTrace\": \"at System.Drawing.Graphics.CopyFromScreen(...)\"}}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "ccddeeff-3344-5566-7788-99aabbccddee" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"ccddeeff-3344-5566-7788-99aabbccddee\",\r\n \"StartTime\": \"2026-06-29T03:30:01.500Z\",\r\n \"EndTime\": \"2026-06-29T03:30:09.220Z\",\r\n \"State\": \"Faulted\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"System.ComponentModel.Win32Exception (6): The handle is invalid.\\r\\n at System.Drawing.Graphics.CopyFromScreen(Int32 sourceX, Int32 sourceY, Int32 destinationX, Int32 destinationY, Size blockRegionSize, CopyPixelOperation copyPixelOperation)\",\r\n \"CreationTime\": \"2026-06-29T03:30:01.400Z\",\r\n \"ReleaseName\": \"DashboardSnapshotBot\",\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"OrganizationUnitId\": 7791650,\r\n \"FolderKey\": \"a7e7f8a9-b0c1-2345-6789-abcdef012345\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVCMON\",\r\n \"OrchestratorUserIdentity\": \"svcmonitor\",\r\n \"LastModificationTime\": \"2026-06-29T03:30:09.220Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"ccddeeff-3344-5566-7788-99aabbccddee\",\r\n \"Id\": 672000011\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "ccddeeff-3344-5566-7788-99aabbccddee" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-29T03:30:09.200Z\", \"Message\": \"Take Screenshot: System.ComponentModel.Win32Exception (6): The handle is invalid. at System.Drawing.Graphics.CopyFromScreen(...)\"},\r\n {\"Level\": \"Error\", \"TimeStamp\": \"2026-06-29T03:30:09.050Z\", \"Message\": \"[Robot] Screen capture requested but no active interactive desktop session is available for this unattended run. CopyFromScreen has no valid device context.\"},\r\n {\"Level\": \"Info\", \"TimeStamp\": \"2026-06-29T03:30:04.100Z\", \"Message\": \"[DashboardSnapshotBot] Capturing dashboard snapshot.\"}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "ccddeeff-3344-5566-7788-99aabbccddee" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\"Activity\": \"Main Sequence\", \"Status\": \"Faulted\", \"StartTime\": \"2026-06-29T03:30:02.010Z\", \"EndTime\": \"2026-06-29T03:30:09.210Z\"},\r\n {\"Activity\": \"Open dashboard\", \"Status\": \"Succeeded\", \"StartTime\": \"2026-06-29T03:30:02.220Z\", \"EndTime\": \"2026-06-29T03:30:08.700Z\"},\r\n {\"Activity\": \"Take Screenshot\", \"Status\": \"Faulted\", \"StartTime\": \"2026-06-29T03:30:08.720Z\", \"EndTime\": \"2026-06-29T03:30:09.200Z\", \"Error\": {\"Type\": \"System.ComponentModel.Win32Exception\", \"Message\": \"The handle is invalid.\", \"Activity\": \"Take Screenshot\", \"StackTrace\": \"at System.Drawing.Graphics.CopyFromScreen(...)\"}}\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "users", + "get", + "bb22cc33-dd44-ee55-ff66-778899aabbcc" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"UserGet\",\r\n \"Data\": {\r\n \"Key\": \"bb22cc33-dd44-ee55-ff66-778899aabbcc\",\r\n \"UserName\": \"svcmonitor\",\r\n \"DomainName\": \"UIPATH\",\r\n \"FullName\": \"Service Account Monitoring\",\r\n \"Type\": \"User\",\r\n \"UnattendedRobot\": {\r\n \"ExecutionSettings\": {\"LoginToConsole\": true, \"RunOnlyOneJobAtATime\": false, \"ResolutionWidth\": 1920, \"ResolutionHeight\": 1080, \"ResolutionDepth\": 32},\r\n \"UserName\": \"UIPATH\\\\SVCMON\",\r\n \"CredentialStoreId\": null,\r\n \"CredentialType\": \"Default\",\r\n \"PasswordLastSet\": \"2026-06-01T09:00:00Z\"\r\n },\r\n \"TenantId\": 1,\r\n \"Id\": 7510\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\", \"Name\": \"original_email@test.com's workspace\", \"Path\": \"original_email@test.com's workspace\", \"Type\": \"Personal\", \"ParentKey\": \"\"},\r\n {\"Key\": \"a7e7f8a9-b0c1-2345-6789-abcdef012345\", \"Name\": \"MonitoringOps\", \"Path\": \"MonitoringOps\", \"Description\": \"Folder hosting the DashboardSnapshotBot unattended process.\", \"Type\": \"Standard\", \"ParentKey\": \"\"},\r\n {\"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\", \"Name\": \"Shared\", \"Path\": \"Shared\", \"Type\": \"Standard\", \"ParentKey\": \"\"}\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/task.yaml index 4268fd1959..393e078071 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-screenshot-handle-invalid/task.yaml @@ -28,21 +28,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/README.md new file mode 100644 index 0000000000..786b711829 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/README.md @@ -0,0 +1,21 @@ +# Scenario: rpa-serverless-robot-units + +Synthetic Orchestrator control-plane scenario. Serverless job `c3930001-3333-4333-8333-333333333333` +(`InvoiceParse`, folder Cloud Robots) goes straight to Faulted with +`Automation cannot be started. Your tenant's assigned Robot Units have been exceeded.` +Maps to `products/orchestrator/playbooks/serverless-license-quota.md`. + +The discriminating evidence: `jobs get` shows `RuntimeType` Serverless, history +is Pending → Faulted (never Running), and the message names a **tenant-scope +Robot Units** limit. `machines list` is empty (serverless jobs bind no listed +machine template — expected, not a no-host fault). The fix is to allocate more +Robot Units to the tenant (or reduce serverless concurrency) — NOT an +executor/session, credential, robot-version, or rerun fix. + +Control-plane failure (pre-workflow), so there is no `process/` source; the +agent diagnoses from `uip or jobs`/`machines` evidence. Grading: +`skill_triggered` + `llm_judge` vs `RESOLUTION.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/0274facfc4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/0274facfc4.json deleted file mode 100644 index fb837a0bff..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/0274facfc4.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "c3930001-3333-4333-8333-333333333333", - "State": "Faulted", - "ProcessName": "InvoiceParse", - "ReleaseName": "InvoiceParse", - "StartTime": "2026-07-19T14:22:11.000Z", - "EndTime": "2026-07-19T14:22:11.300Z", - "CreationTime": "2026-07-19T14:22:10.700Z", - "HostMachineName": null, - "Type": "Unattended", - "RuntimeType": "Serverless", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/2e3900c716.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/2e3900c716.json deleted file mode 100644 index fb837a0bff..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/2e3900c716.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "c3930001-3333-4333-8333-333333333333", - "State": "Faulted", - "ProcessName": "InvoiceParse", - "ReleaseName": "InvoiceParse", - "StartTime": "2026-07-19T14:22:11.000Z", - "EndTime": "2026-07-19T14:22:11.300Z", - "CreationTime": "2026-07-19T14:22:10.700Z", - "HostMachineName": null, - "Type": "Unattended", - "RuntimeType": "Serverless", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/3479649ec5.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/3479649ec5.json deleted file mode 100644 index 0449455218..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/3479649ec5.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T14:22:11.250Z", - "Message": "Automation cannot be started. Your tenant's assigned Robot Units have been exceeded. Allocate more Robot Units to your tenant to continue." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/75949d659e.json deleted file mode 100644 index fb837a0bff..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/75949d659e.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "c3930001-3333-4333-8333-333333333333", - "State": "Faulted", - "ProcessName": "InvoiceParse", - "ReleaseName": "InvoiceParse", - "StartTime": "2026-07-19T14:22:11.000Z", - "EndTime": "2026-07-19T14:22:11.300Z", - "CreationTime": "2026-07-19T14:22:10.700Z", - "HostMachineName": null, - "Type": "Unattended", - "RuntimeType": "Serverless", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/8883fea2d6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/8883fea2d6.json deleted file mode 100644 index 89ef180924..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/8883fea2d6.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/8b38fe7f7c.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/8b38fe7f7c.json deleted file mode 100644 index 2283defd04..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/8b38fe7f7c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-19T14:22:11.300Z", - "JobId": 671550303 - }, - { - "State": "Pending", - "Timestamp": "2026-07-19T14:22:10.740Z", - "JobId": 671550303 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/README.md deleted file mode 100644 index b44e928b4d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Scenario: rpa-serverless-robot-units - -Synthetic Orchestrator control-plane scenario. Serverless job `c3930001-3333-4333-8333-333333333333` -(`InvoiceParse`, folder Cloud Robots) goes straight to Faulted with -`Automation cannot be started. Your tenant's assigned Robot Units have been exceeded.` -Maps to `products/orchestrator/playbooks/serverless-license-quota.md`. - -The discriminating evidence: `jobs get` shows `RuntimeType` Serverless, history -is Pending → Faulted (never Running), and the message names a **tenant-scope -Robot Units** limit. `machines list` is empty (serverless jobs bind no listed -machine template — expected, not a no-host fault). The fix is to allocate more -Robot Units to the tenant (or reduce serverless concurrency) — NOT an -executor/session, credential, robot-version, or rerun fix. - -Control-plane failure (pre-workflow), so there is no `process/` source; the -agent diagnoses from `uip or jobs`/`machines` evidence. Grading: -`skill_triggered` + `llm_judge` vs `RESOLUTION.md`. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/d76e16aab8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/d76e16aab8.json deleted file mode 100644 index 0449455218..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/d76e16aab8.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T14:22:11.250Z", - "Message": "Automation cannot be started. Your tenant's assigned Robot Units have been exceeded. Allocate more Robot Units to your tenant to continue." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/dc064823f4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/dc064823f4.json deleted file mode 100644 index 89ef180924..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/dc064823f4.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [], - "Pagination": { - "Returned": 0, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/e825135ddc.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/e825135ddc.json deleted file mode 100644 index a0140eceb2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/e825135ddc.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3930001-3333-4333-8333-333333333333", - "StartTime": "2026-07-19T14:22:11.000Z", - "EndTime": "2026-07-19T14:22:11.300Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Manual", - "SourceType": "Manual", - "Info": "Automation cannot be started. Your tenant's assigned Robot Units have been exceeded. Allocate more Robot Units to your tenant to continue.", - "CreationTime": "2026-07-19T14:22:10.700Z", - "ReleaseName": "InvoiceParse", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": null, - "RuntimeType": "Serverless", - "RequiresUserInteraction": false, - "FolderKey": "f3a30003-0000-4000-8000-000000000003", - "ProcessType": "Process", - "LastModificationTime": "2026-07-19T14:22:11.300Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3930001333343338333333333333333", - "Id": 671550303 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/ee2628e6b6.json deleted file mode 100644 index 4a4e56f3b4..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f3a30003-0000-4000-8000-000000000003", - "Name": "Cloud Robots", - "Path": "Cloud Robots", - "Description": "Serverless cloud-robot automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/f5f2f217c8.json deleted file mode 100644 index fb837a0bff..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "c3930001-3333-4333-8333-333333333333", - "State": "Faulted", - "ProcessName": "InvoiceParse", - "ReleaseName": "InvoiceParse", - "StartTime": "2026-07-19T14:22:11.000Z", - "EndTime": "2026-07-19T14:22:11.300Z", - "CreationTime": "2026-07-19T14:22:10.700Z", - "HostMachineName": null, - "Type": "Unattended", - "RuntimeType": "Serverless", - "Source": "Manual", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/manifest.json deleted file mode 100644 index 198f0634ad..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/m/r/manifest.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-serverless-robot-units scenario. The agent chains folders list -> jobs list --state Faulted -> jobs get -> jobs logs -> jobs history. Discriminating evidence: the job is RuntimeType Serverless with history Pending->Faulted (never Running) and the explicit message 'Your tenant's assigned Robot Units have been exceeded' — a tenant-scope licensing/quota limit, not an executor/session/credential fault. machines list is empty (serverless jobs do not bind a listed machine template). Specific rules precede generic ones (first-match-wins).", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key f3a30003-0000-4000-8000-000000000003 --state Faulted", "file": "0274facfc4.json" }, - { "match": "or jobs list --folder-key f3a30003-0000-4000-8000-000000000003", "file": "2e3900c716.json" }, - { "match": "or jobs list --state Faulted", "file": "f5f2f217c8.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs get c3930001-3333-4333-8333-333333333333", "file": "e825135ddc.json" }, - { "match": "or jobs logs c3930001-3333-4333-8333-333333333333 --level Error", "file": "3479649ec5.json" }, - { "match": "or jobs logs c3930001-3333-4333-8333-333333333333", "file": "d76e16aab8.json" }, - { "match": "or jobs history c3930001-3333-4333-8333-333333333333", "file": "8b38fe7f7c.json" }, - { "match": "or machines list --all-fields", "file": "8883fea2d6.json" }, - { "match": "or machines list", "file": "dc064823f4.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/uip-fixture.json new file mode 100644 index 0000000000..39c9346a9e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/data/uip-fixture.json @@ -0,0 +1,134 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f3a30003-0000-4000-8000-000000000003", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3930001-3333-4333-8333-333333333333\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceParse\",\r\n \"ReleaseName\": \"InvoiceParse\",\r\n \"StartTime\": \"2026-07-19T14:22:11.000Z\",\r\n \"EndTime\": \"2026-07-19T14:22:11.300Z\",\r\n \"CreationTime\": \"2026-07-19T14:22:10.700Z\",\r\n \"HostMachineName\": null,\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Serverless\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3930001-3333-4333-8333-333333333333", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T14:22:11.250Z\",\r\n \"Message\": \"Automation cannot be started. Your tenant's assigned Robot Units have been exceeded. Allocate more Robot Units to your tenant to continue.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f3a30003-0000-4000-8000-000000000003" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3930001-3333-4333-8333-333333333333\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceParse\",\r\n \"ReleaseName\": \"InvoiceParse\",\r\n \"StartTime\": \"2026-07-19T14:22:11.000Z\",\r\n \"EndTime\": \"2026-07-19T14:22:11.300Z\",\r\n \"CreationTime\": \"2026-07-19T14:22:10.700Z\",\r\n \"HostMachineName\": null,\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Serverless\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3930001-3333-4333-8333-333333333333\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceParse\",\r\n \"ReleaseName\": \"InvoiceParse\",\r\n \"StartTime\": \"2026-07-19T14:22:11.000Z\",\r\n \"EndTime\": \"2026-07-19T14:22:11.300Z\",\r\n \"CreationTime\": \"2026-07-19T14:22:10.700Z\",\r\n \"HostMachineName\": null,\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Serverless\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3930001-3333-4333-8333-333333333333" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3930001-3333-4333-8333-333333333333\",\r\n \"StartTime\": \"2026-07-19T14:22:11.000Z\",\r\n \"EndTime\": \"2026-07-19T14:22:11.300Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Automation cannot be started. Your tenant's assigned Robot Units have been exceeded. Allocate more Robot Units to your tenant to continue.\",\r\n \"CreationTime\": \"2026-07-19T14:22:10.700Z\",\r\n \"ReleaseName\": \"InvoiceParse\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": null,\r\n \"RuntimeType\": \"Serverless\",\r\n \"RequiresUserInteraction\": false,\r\n \"FolderKey\": \"f3a30003-0000-4000-8000-000000000003\",\r\n \"ProcessType\": \"Process\",\r\n \"LastModificationTime\": \"2026-07-19T14:22:11.300Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3930001333343338333333333333333\",\r\n \"Id\": 671550303\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c3930001-3333-4333-8333-333333333333" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T14:22:11.250Z\",\r\n \"Message\": \"Automation cannot be started. Your tenant's assigned Robot Units have been exceeded. Allocate more Robot Units to your tenant to continue.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "c3930001-3333-4333-8333-333333333333" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-19T14:22:11.300Z\",\r\n \"JobId\": 671550303\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-19T14:22:10.740Z\",\r\n \"JobId\": 671550303\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--all-fields" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f3a30003-0000-4000-8000-000000000003\",\r\n \"Name\": \"Cloud Robots\",\r\n \"Path\": \"Cloud Robots\",\r\n \"Description\": \"Serverless cloud-robot automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c3930001-3333-4333-8333-333333333333\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceParse\",\r\n \"ReleaseName\": \"InvoiceParse\",\r\n \"StartTime\": \"2026-07-19T14:22:11.000Z\",\r\n \"EndTime\": \"2026-07-19T14:22:11.300Z\",\r\n \"CreationTime\": \"2026-07-19T14:22:10.700Z\",\r\n \"HostMachineName\": null,\r\n \"Type\": \"Unattended\",\r\n \"RuntimeType\": \"Serverless\",\r\n \"Source\": \"Manual\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [],\r\n \"Pagination\": {\r\n \"Returned\": 0,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/task.yaml index e986db9c84..7426eb2ef7 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-serverless-robot-units/task.yaml @@ -17,20 +17,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/README.md new file mode 100644 index 0000000000..3a7635d63e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/README.md @@ -0,0 +1,71 @@ +# Session-Creation Timeout — Host Latency Exceeds the Window + +This scenario reproduces the **"Could Not Start Executor — Creating User +Session Timed Out"** playbook: an unattended job enters Running, the Robot +begins creating the Windows session, the create-session call **times out +(~120s)**, and the job faults with: + +``` +Could not start executor. Creating user session timed out. +``` + +Nothing refused the credential — session creation simply took longer than the +timeout window (a host-side latency / resource / infrastructure condition). + +## What this scenario uncovers + +**Root Cause:** Three identical `NightlyReconciler` runs faulted on +`RECON-BOT-01` (keys `c0ffee01-...` at 02:00Z, `c0ffee02-...` at 01:00Z, +`c0ffee03-...` at 00:00Z), each after ~120s in Running. The robot log states +the create-session call timed out and that LSA returned **no** logon rejection +— a pure session-creation timeout, not a credential or no-host fault. The fix +is to raise the session-creation timeout (`UIPATH_SESSION_TIMEOUT`) and reduce +host-side logon/resource latency — **not** a Robot-version upgrade (there is no +version-specific fix for this error). + +Maps to: +`references/products/orchestrator/playbooks/job-faulted-session-timeout.md`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | minimal background-unattended UiPath project (LogMessage + Delay) | + +> **Note on fixtures.** These fixtures were authored from the documented +> playbook signature, not captured from a real `.local/investigations/` +> session. Regenerate from a real failed-job session before treating this +> test's score as a hard regression signal. + +## How this differs from the sibling "Could not start executor" playbook + +`job-faulted-logon-failure` and this playbook both start with +`Could not start executor`. The agent must pick the right one: + +| Signal | `job-faulted-logon-failure` | `job-faulted-session-timeout` *(this)* | +|---|---|---| +| Error phrasing | `Logon failed for user` / `account is locked` / `RDP connection failed` | `Creating user session timed out` | +| Windows code | `0x0000052E` / `0x00000775` / `0x00000532` / `131092` | none — a pure timeout | +| LSA verdict | active rejection | no rejection; create-session call timed out | +| Duration | sub-second (immediate refusal) | ~ the session-creation timeout (~120s) | +| Discriminating fix | credential / account / RDP | raise `UIPATH_SESSION_TIMEOUT` + reduce host latency | + +`no-host` / stuck-Pending is also ruled out: `jobs history` shows the job +reached **Running** before faulting, and a runtime is connected on the host. + +## Success criteria + +Scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent matched `orchestrator/playbooks/job-faulted-session-timeout.md` and + identified a **session-creation timeout** (create-session exceeded the ~120s + window; LSA returned no rejection) as the root cause — not a logon failure, + not no-host, and not a Robot-version defect. +- Conclusion proposes the playbook's resolution: **raise `UIPATH_SESSION_TIMEOUT` + and reduce host-side logon/resource latency** (re-run as interim mitigation) — + not a Robot-version upgrade. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/README.md deleted file mode 100644 index 2437e67e0b..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/README.md +++ /dev/null @@ -1,70 +0,0 @@ -# Session-Creation Timeout — Host Latency Exceeds the Window - -This scenario reproduces the **"Could Not Start Executor — Creating User -Session Timed Out"** playbook: an unattended job enters Running, the Robot -begins creating the Windows session, the create-session call **times out -(~120s)**, and the job faults with: - -``` -Could not start executor. Creating user session timed out. -``` - -Nothing refused the credential — session creation simply took longer than the -timeout window (a host-side latency / resource / infrastructure condition). - -## What this scenario uncovers - -**Root Cause:** Three identical `NightlyReconciler` runs faulted on -`RECON-BOT-01` (keys `c0ffee01-...` at 02:00Z, `c0ffee02-...` at 01:00Z, -`c0ffee03-...` at 00:00Z), each after ~120s in Running. The robot log states -the create-session call timed out and that LSA returned **no** logon rejection -— a pure session-creation timeout, not a credential or no-host fault. The fix -is to raise the session-creation timeout (`UIPATH_SESSION_TIMEOUT`) and reduce -host-side logon/resource latency — **not** a Robot-version upgrade (there is no -version-specific fix for this error). - -Maps to: -`references/products/orchestrator/playbooks/job-faulted-session-timeout.md`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` + `m/uip.cmd` | shared from `../../../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | minimal background-unattended UiPath project (LogMessage + Delay) | -| `data/m/r/*.json` | **synthetic** canned `uip` responses authored from the playbook signature | -| `data/m/r/manifest.json` | dispatch table mapping each command pattern to its fixture | - -> **Note on fixtures.** These fixtures were authored from the documented -> playbook signature, not captured from a real `.local/investigations/` -> session. Regenerate from a real failed-job session before treating this -> test's score as a hard regression signal. - -## How this differs from the sibling "Could not start executor" playbook - -`job-faulted-logon-failure` and this playbook both start with -`Could not start executor`. The agent must pick the right one: - -| Signal | `job-faulted-logon-failure` | `job-faulted-session-timeout` *(this)* | -|---|---|---| -| Error phrasing | `Logon failed for user` / `account is locked` / `RDP connection failed` | `Creating user session timed out` | -| Windows code | `0x0000052E` / `0x00000775` / `0x00000532` / `131092` | none — a pure timeout | -| LSA verdict | active rejection | no rejection; create-session call timed out | -| Duration | sub-second (immediate refusal) | ~ the session-creation timeout (~120s) | -| Discriminating fix | credential / account / RDP | raise `UIPATH_SESSION_TIMEOUT` + reduce host latency | - -`no-host` / stuck-Pending is also ruled out: `jobs history` shows the job -reached **Running** before faulting, and a runtime is connected on the host. - -## Success criteria - -Scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent matched `orchestrator/playbooks/job-faulted-session-timeout.md` and - identified a **session-creation timeout** (create-session exceeded the ~120s - window; LSA returned no rejection) as the root cause — not a logon failure, - not no-host, and not a Robot-version defect. -- Conclusion proposes the playbook's resolution: **raise `UIPATH_SESSION_TIMEOUT` - and reduce host-side logon/resource latency** (re-run as interim mitigation) — - not a Robot-version upgrade. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/folders.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/folders.json deleted file mode 100644 index 611030b58a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/folders.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "b2c8cb22-2ebc-4c27-87f3-05efe701af1c", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a1b2c3d4-e5f6-4789-abcd-000000000001", - "Name": "UnattendedOps", - "Path": "UnattendedOps", - "Description": "Folder hosting the NightlyReconciler background unattended process.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "e0772cbe-c19c-4d5f-926c-043c2d2d300a", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-get.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-get.json deleted file mode 100644 index bb17690aa9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-get.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c0ffee01-1111-4222-8333-444455556666", - "StartTime": "2026-07-13T02:00:05.100Z", - "EndTime": "2026-07-13T02:02:05.400Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "Schedule", - "BatchExecutionKey": "7f3a2b10-9c8d-4e6f-a1b2-c3d4e5f60718", - "Info": "Could not start executor. Creating user session timed out.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-07-13T02:00:04.900Z", - "ReleaseName": "NightlyReconciler", - "Type": "Unattended", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "RECON-BOT-01", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 5120441, - "OrganizationUnitId": 7791617, - "FolderKey": "a1b2c3d4-e5f6-4789-abcd-000000000001", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\SVC_RECON", - "OrchestratorUserIdentity": "svc_recon", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-07-13T02:02:05.400Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c0ffee01-1111-4222-8333-444455556666", - "Id": 671220455 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-history.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-history.json deleted file mode 100644 index e94fcca6a7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-history.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Pending", - "CreationTime": "2026-07-13T02:00:04.900Z", - "Reason": "Job created by schedule trigger 'NightlyReconciler-0200'." - }, - { - "State": "Running", - "CreationTime": "2026-07-13T02:00:05.100Z", - "Reason": "Robot on RECON-BOT-01 accepted the job; starting executor." - }, - { - "State": "Faulted", - "CreationTime": "2026-07-13T02:02:05.400Z", - "Reason": "Could not start executor. Creating user session timed out." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-logs.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-logs.json deleted file mode 100644 index fea9bb2a48..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/job-logs.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-13T02:02:05.390Z", - "Message": "Job 'c0ffee01-1111-4222-8333-444455556666' failed to start. Could not start executor. Creating user session timed out.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)" - }, - { - "Level": "Error", - "TimeStamp": "2026-07-13T02:02:05.210Z", - "Message": "[Robot] Creating a Windows session for UIPATH\\SVC_RECON on RECON-BOT-01 did not complete within the 120s session-creation timeout. The create-session call timed out; LSA returned NO logon rejection (no bad-password, locked, or expired status)." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/jobs-faulted.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/jobs-faulted.json deleted file mode 100644 index 3c92419cf9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/jobs-faulted.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c0ffee01-1111-4222-8333-444455556666", - "State": "Faulted", - "ProcessName": "NightlyReconciler", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "RECON-BOT-01", - "MachineKey": "beef0001-2222-4333-8444-555566667777", - "StartTime": "2026-07-13T02:00:05.100Z", - "EndTime": "2026-07-13T02:02:05.400Z", - "CreationTime": "2026-07-13T02:00:04.900Z" - }, - { - "Key": "c0ffee02-2222-4333-8444-555566667788", - "State": "Faulted", - "ProcessName": "NightlyReconciler", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "RECON-BOT-01", - "MachineKey": "beef0001-2222-4333-8444-555566667777", - "StartTime": "2026-07-13T01:00:05.050Z", - "EndTime": "2026-07-13T01:02:05.300Z", - "CreationTime": "2026-07-13T01:00:04.870Z" - }, - { - "Key": "c0ffee03-3333-4444-8555-666677778899", - "State": "Faulted", - "ProcessName": "NightlyReconciler", - "ReleaseKey": "", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "RECON-BOT-01", - "MachineKey": "beef0001-2222-4333-8444-555566667777", - "StartTime": "2026-07-13T00:00:05.020Z", - "EndTime": "2026-07-13T00:02:05.260Z", - "CreationTime": "2026-07-13T00:00:04.800Z" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/machines-all.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/machines-all.json deleted file mode 100644 index 08d22918f2..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/machines-all.json +++ /dev/null @@ -1,74 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "licenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2062", - "name": "RECON-BOT-01", - "description": "Dedicated unattended host for NightlyReconciler.", - "type": "Template", - "scope": "Default", - "nonProductionSlots": 0, - "unattendedSlots": 1, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "beef0001-2222-4333-8444-555566667777", - "endpointDetectionStatus": "Available", - "robotVersions": [ - { - "count": 1, - "version": "23.4.7" - } - ], - "robotUsers": [ - { - "userName": "UIPATH\\SVC_RECON" - } - ], - "automationType": "Any", - "targetFramework": "Windows", - "tags": [], - "id": 1990321 - }, - { - "name": "original_email@test.com's workspace machine", - "type": "Template", - "scope": "PersonalWorkspace", - "nonProductionSlots": 0, - "unattendedSlots": 1, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "6f12f3e7-75f5-475e-aa34-3f5bd5c564e1", - "endpointDetectionStatus": "NotAvailable", - "robotVersions": [ - { - "count": 1, - "version": "26.0.193-cloud.23059" - } - ], - "robotUsers": [], - "automationType": "Any", - "targetFramework": "Any", - "tags": [], - "id": 961460 - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/machines.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/machines.json deleted file mode 100644 index 26ef99e5b0..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/machines.json +++ /dev/null @@ -1,50 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "Name": "RECON-BOT-01", - "ID": 1990321, - "Key": "beef0001-2222-4333-8444-555566667777", - "Type": "Template", - "Scope": "Default", - "Description": "Dedicated unattended host for NightlyReconciler.", - "UnattendedSlots": 1, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2062", - "robotVersions": [ - { - "count": 1, - "version": "23.4.7" - } - ] - }, - { - "Name": "original_email@test.com's workspace machine", - "ID": 961460, - "Key": "6f12f3e7-75f5-475e-aa34-3f5bd5c564e1", - "Type": "Template", - "Scope": "PersonalWorkspace", - "Description": "", - "UnattendedSlots": 1, - "HeadlessSlots": 0, - "NonProductionSlots": 0, - "TestAutomationSlots": 0, - "LicenseKey": "", - "robotVersions": [ - { - "count": 1, - "version": "26.0.193-cloud.23059" - } - ] - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/manifest.json deleted file mode 100644 index 049999ecf9..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/m/r/manifest.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-session-creation-timeout scenario (job-faulted-session-timeout playbook). The agent derives keys by chaining: folders list -> jobs list --state Faulted -> jobs get -> jobs logs -> jobs history -> machines list. The discriminating evidence is the 'Creating user session timed out' string with NO logon-failure code (rules out job-faulted-logon-failure) plus a ~120s Running-then-Faulted history (rules out no-host/Pending) — a session-creation timeout from host latency. The Robot version 23.4.7 on RECON-BOT-01 is a data point, NOT the cause (no version-specific fix). Specific rules precede generic ones so first-match-wins picks the narrower fixture.", - "rules": [ - { - "match": "or folders list", - "file": "folders.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-e5f6-4789-abcd-000000000001 --state Faulted", - "file": "jobs-faulted.json" - }, - { - "match": "or jobs list --folder-key a1b2c3d4-e5f6-4789-abcd-000000000001", - "file": "jobs-faulted.json" - }, - { - "match": "or jobs get c0ffee01-1111-4222-8333-444455556666", - "file": "job-get.json" - }, - { - "match": "or jobs logs c0ffee01-1111-4222-8333-444455556666 --level Error", - "file": "job-logs.json" - }, - { - "match": "or jobs logs c0ffee01-1111-4222-8333-444455556666", - "file": "job-logs.json" - }, - { - "match": "or jobs history c0ffee01-1111-4222-8333-444455556666", - "file": "job-history.json" - }, - { - "match": "or machines list --all-fields", - "file": "machines-all.json" - }, - { - "match": "or machines list", - "file": "machines.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Proxy any docsai query to the real uip CLI; query strings vary between runs so canned responses go stale. Optional corroboration step in the playbook - an empty/failed KB result is not disconfirming." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched above returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/uip-fixture.json new file mode 100644 index 0000000000..a613f67897 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/data/uip-fixture.json @@ -0,0 +1,112 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-e5f6-4789-abcd-000000000001", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c0ffee01-1111-4222-8333-444455556666\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReconciler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"RECON-BOT-01\",\r\n \"MachineKey\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"StartTime\": \"2026-07-13T02:00:05.100Z\",\r\n \"EndTime\": \"2026-07-13T02:02:05.400Z\",\r\n \"CreationTime\": \"2026-07-13T02:00:04.900Z\"\r\n },\r\n {\r\n \"Key\": \"c0ffee02-2222-4333-8444-555566667788\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReconciler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"RECON-BOT-01\",\r\n \"MachineKey\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"StartTime\": \"2026-07-13T01:00:05.050Z\",\r\n \"EndTime\": \"2026-07-13T01:02:05.300Z\",\r\n \"CreationTime\": \"2026-07-13T01:00:04.870Z\"\r\n },\r\n {\r\n \"Key\": \"c0ffee03-3333-4444-8555-666677778899\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReconciler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"RECON-BOT-01\",\r\n \"MachineKey\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"StartTime\": \"2026-07-13T00:00:05.020Z\",\r\n \"EndTime\": \"2026-07-13T00:02:05.260Z\",\r\n \"CreationTime\": \"2026-07-13T00:00:04.800Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c0ffee01-1111-4222-8333-444455556666", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-13T02:02:05.390Z\",\r\n \"Message\": \"Job 'c0ffee01-1111-4222-8333-444455556666' failed to start. Could not start executor. Creating user session timed out.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-13T02:02:05.210Z\",\r\n \"Message\": \"[Robot] Creating a Windows session for UIPATH\\\\SVC_RECON on RECON-BOT-01 did not complete within the 120s session-creation timeout. The create-session call timed out; LSA returned NO logon rejection (no bad-password, locked, or expired status).\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "a1b2c3d4-e5f6-4789-abcd-000000000001" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c0ffee01-1111-4222-8333-444455556666\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReconciler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"RECON-BOT-01\",\r\n \"MachineKey\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"StartTime\": \"2026-07-13T02:00:05.100Z\",\r\n \"EndTime\": \"2026-07-13T02:02:05.400Z\",\r\n \"CreationTime\": \"2026-07-13T02:00:04.900Z\"\r\n },\r\n {\r\n \"Key\": \"c0ffee02-2222-4333-8444-555566667788\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReconciler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"RECON-BOT-01\",\r\n \"MachineKey\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"StartTime\": \"2026-07-13T01:00:05.050Z\",\r\n \"EndTime\": \"2026-07-13T01:02:05.300Z\",\r\n \"CreationTime\": \"2026-07-13T01:00:04.870Z\"\r\n },\r\n {\r\n \"Key\": \"c0ffee03-3333-4444-8555-666677778899\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"NightlyReconciler\",\r\n \"ReleaseKey\": \"\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"RECON-BOT-01\",\r\n \"MachineKey\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"StartTime\": \"2026-07-13T00:00:05.020Z\",\r\n \"EndTime\": \"2026-07-13T00:02:05.260Z\",\r\n \"CreationTime\": \"2026-07-13T00:00:04.800Z\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c0ffee01-1111-4222-8333-444455556666" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c0ffee01-1111-4222-8333-444455556666\",\r\n \"StartTime\": \"2026-07-13T02:00:05.100Z\",\r\n \"EndTime\": \"2026-07-13T02:02:05.400Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"BatchExecutionKey\": \"7f3a2b10-9c8d-4e6f-a1b2-c3d4e5f60718\",\r\n \"Info\": \"Could not start executor. Creating user session timed out.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-07-13T02:00:04.900Z\",\r\n \"ReleaseName\": \"NightlyReconciler\",\r\n \"Type\": \"Unattended\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"RECON-BOT-01\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 5120441,\r\n \"OrganizationUnitId\": 7791617,\r\n \"FolderKey\": \"a1b2c3d4-e5f6-4789-abcd-000000000001\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_RECON\",\r\n \"OrchestratorUserIdentity\": \"svc_recon\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-07-13T02:02:05.400Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c0ffee01-1111-4222-8333-444455556666\",\r\n \"Id\": 671220455\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "c0ffee01-1111-4222-8333-444455556666" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-13T02:02:05.390Z\",\r\n \"Message\": \"Job 'c0ffee01-1111-4222-8333-444455556666' failed to start. Could not start executor. Creating user session timed out.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-13T02:02:05.210Z\",\r\n \"Message\": \"[Robot] Creating a Windows session for UIPATH\\\\SVC_RECON on RECON-BOT-01 did not complete within the 120s session-creation timeout. The create-session call timed out; LSA returned NO logon rejection (no bad-password, locked, or expired status).\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "c0ffee01-1111-4222-8333-444455556666" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Pending\",\r\n \"CreationTime\": \"2026-07-13T02:00:04.900Z\",\r\n \"Reason\": \"Job created by schedule trigger 'NightlyReconciler-0200'.\"\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"CreationTime\": \"2026-07-13T02:00:05.100Z\",\r\n \"Reason\": \"Robot on RECON-BOT-01 accepted the job; starting executor.\"\r\n },\r\n {\r\n \"State\": \"Faulted\",\r\n \"CreationTime\": \"2026-07-13T02:02:05.400Z\",\r\n \"Reason\": \"Could not start executor. Creating user session timed out.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--all-fields" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"licenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2062\",\r\n \"name\": \"RECON-BOT-01\",\r\n \"description\": \"Dedicated unattended host for NightlyReconciler.\",\r\n \"type\": \"Template\",\r\n \"scope\": \"Default\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 1,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"endpointDetectionStatus\": \"Available\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"23.4.7\"\r\n }\r\n ],\r\n \"robotUsers\": [\r\n {\r\n \"userName\": \"UIPATH\\\\SVC_RECON\"\r\n }\r\n ],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Windows\",\r\n \"tags\": [],\r\n \"id\": 1990321\r\n },\r\n {\r\n \"name\": \"original_email@test.com's workspace machine\",\r\n \"type\": \"Template\",\r\n \"scope\": \"PersonalWorkspace\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 1,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"6f12f3e7-75f5-475e-aa34-3f5bd5c564e1\",\r\n \"endpointDetectionStatus\": \"NotAvailable\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"26.0.193-cloud.23059\"\r\n }\r\n ],\r\n \"robotUsers\": [],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Any\",\r\n \"tags\": [],\r\n \"id\": 961460\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b2c8cb22-2ebc-4c27-87f3-05efe701af1c\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a1b2c3d4-e5f6-4789-abcd-000000000001\",\r\n \"Name\": \"UnattendedOps\",\r\n \"Path\": \"UnattendedOps\",\r\n \"Description\": \"Folder hosting the NightlyReconciler background unattended process.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"e0772cbe-c19c-4d5f-926c-043c2d2d300a\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"Name\": \"RECON-BOT-01\",\r\n \"ID\": 1990321,\r\n \"Key\": \"beef0001-2222-4333-8444-555566667777\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"Default\",\r\n \"Description\": \"Dedicated unattended host for NightlyReconciler.\",\r\n \"UnattendedSlots\": 1,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2062\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"23.4.7\"\r\n }\r\n ]\r\n },\r\n {\r\n \"Name\": \"original_email@test.com's workspace machine\",\r\n \"ID\": 961460,\r\n \"Key\": \"6f12f3e7-75f5-475e-aa34-3f5bd5c564e1\",\r\n \"Type\": \"Template\",\r\n \"Scope\": \"PersonalWorkspace\",\r\n \"Description\": \"\",\r\n \"UnattendedSlots\": 1,\r\n \"HeadlessSlots\": 0,\r\n \"NonProductionSlots\": 0,\r\n \"TestAutomationSlots\": 0,\r\n \"LicenseKey\": \"\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"26.0.193-cloud.23059\"\r\n }\r\n ]\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/task.yaml index 7f0ab3285b..dcc64b2dc2 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-session-creation-timeout/task.yaml @@ -27,21 +27,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/README.md new file mode 100644 index 0000000000..d793fd1b7d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/README.md @@ -0,0 +1,22 @@ +# Scenario: rpa-workstation-in-use + +Synthetic Orchestrator control-plane scenario. Job `a1710001-1111-4111-8111-111111111111` +(`DailyExtract`, folder Unattended Ops) faults at executor start with +`Could not start executor. The workstation is in use by another user.` Maps to +`products/orchestrator/playbooks/workstation-in-use-machine-slots.md`. + +The discriminating evidence is in `machines list --all-fields` + `jobs list`: +WS-CLIENT-07 is a Windows 11 client (single interactive session) whose template +grants `unattendedSlots: 3`, and a concurrent job (`HourlySync`) was Running on +the same host when `DailyExtract` was dispatched — so Windows had no second +session to seat it. Non-overlapping runs succeed. The fix is to align the slot +count with the host's capacity (reduce to 1, or use a multi-session host) — NOT +a credential fix, a rerun, or a licensing change. + +Control-plane failure (pre-workflow), so there is no `process/` source; the +agent diagnoses from `uip or jobs`/`machines` evidence. Grading: +`skill_triggered` + `llm_judge` vs `RESOLUTION.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/356bfb7143.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/356bfb7143.json deleted file mode 100644 index 74c1ddb53d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/356bfb7143.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T06:00:03.820Z", - "Message": "Could not start executor. The workstation is in use by another user. Please retry after the user logs off or disconnects." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/4b9e5c8cb4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/4b9e5c8cb4.json deleted file mode 100644 index b42bf9e152..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/4b9e5c8cb4.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "a1710001-1111-4111-8111-111111111111", - "State": "Faulted", - "ProcessName": "DailyExtract", - "ReleaseName": "DailyExtract", - "StartTime": "2026-07-19T06:00:03.100Z", - "EndTime": "2026-07-19T06:00:03.900Z", - "CreationTime": "2026-07-19T06:00:02.500Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "a1710002-2222-4222-8222-222222222222", - "State": "Running", - "ProcessName": "HourlySync", - "ReleaseName": "HourlySync", - "StartTime": "2026-07-19T06:00:01.400Z", - "EndTime": null, - "CreationTime": "2026-07-19T06:00:00.900Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "a1710003-3333-4333-8333-333333333333", - "State": "Successful", - "ProcessName": "DailyExtract", - "ReleaseName": "DailyExtract", - "StartTime": "2026-07-18T06:00:02.700Z", - "EndTime": "2026-07-18T06:04:55.100Z", - "CreationTime": "2026-07-18T06:00:02.100Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/75949d659e.json deleted file mode 100644 index b42bf9e152..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/75949d659e.json +++ /dev/null @@ -1,45 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "a1710001-1111-4111-8111-111111111111", - "State": "Faulted", - "ProcessName": "DailyExtract", - "ReleaseName": "DailyExtract", - "StartTime": "2026-07-19T06:00:03.100Z", - "EndTime": "2026-07-19T06:00:03.900Z", - "CreationTime": "2026-07-19T06:00:02.500Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "a1710002-2222-4222-8222-222222222222", - "State": "Running", - "ProcessName": "HourlySync", - "ReleaseName": "HourlySync", - "StartTime": "2026-07-19T06:00:01.400Z", - "EndTime": null, - "CreationTime": "2026-07-19T06:00:00.900Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - }, - { - "Key": "a1710003-3333-4333-8333-333333333333", - "State": "Successful", - "ProcessName": "DailyExtract", - "ReleaseName": "DailyExtract", - "StartTime": "2026-07-18T06:00:02.700Z", - "EndTime": "2026-07-18T06:04:55.100Z", - "CreationTime": "2026-07-18T06:00:02.100Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/7a6e65f01a.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/7a6e65f01a.json deleted file mode 100644 index 74c1ddb53d..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/7a6e65f01a.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T06:00:03.820Z", - "Message": "Could not start executor. The workstation is in use by another user. Please retry after the user logs off or disconnects." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/8883fea2d6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/8883fea2d6.json deleted file mode 100644 index 9198f958f8..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/8883fea2d6.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "licenseKey": "605b37db-dc58-460f-99ed-93fcfb2f2701", - "name": "WS-CLIENT-07", - "description": "Windows 11 client workstation, dedicated unattended host.", - "type": "Template", - "scope": "Default", - "nonProductionSlots": 0, - "unattendedSlots": 3, - "headlessSlots": 0, - "testAutomationSlots": 0, - "hostingSlots": 0, - "appTestSlots": 0, - "performanceTestSlots": 0, - "agentSlots": 0, - "automationCloudSlots": 0, - "automationCloudTestAutomationSlots": 0, - "key": "beef1001-2222-4333-8444-555566667701", - "endpointDetectionStatus": "Available", - "robotVersions": [ - { - "count": 1, - "version": "24.10.4" - } - ], - "robotUsers": [ - { - "userName": "UIPATH\\SVC_EXTRACT" - } - ], - "automationType": "Any", - "targetFramework": "Windows", - "tags": [], - "id": 1990701 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/8b13f09c9d.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/8b13f09c9d.json deleted file mode 100644 index 822f79ab28..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/8b13f09c9d.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-19T06:00:03.900Z", - "JobId": 671330101 - }, - { - "State": "Running", - "Timestamp": "2026-07-19T06:00:03.110Z", - "JobId": 671330101 - }, - { - "State": "Pending", - "Timestamp": "2026-07-19T06:00:02.540Z", - "JobId": 671330101 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/README.md deleted file mode 100644 index 23655b15fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/README.md +++ /dev/null @@ -1,18 +0,0 @@ -# Scenario: rpa-workstation-in-use - -Synthetic Orchestrator control-plane scenario. Job `a1710001-1111-4111-8111-111111111111` -(`DailyExtract`, folder Unattended Ops) faults at executor start with -`Could not start executor. The workstation is in use by another user.` Maps to -`products/orchestrator/playbooks/workstation-in-use-machine-slots.md`. - -The discriminating evidence is in `machines list --all-fields` + `jobs list`: -WS-CLIENT-07 is a Windows 11 client (single interactive session) whose template -grants `unattendedSlots: 3`, and a concurrent job (`HourlySync`) was Running on -the same host when `DailyExtract` was dispatched — so Windows had no second -session to seat it. Non-overlapping runs succeed. The fix is to align the slot -count with the host's capacity (reduce to 1, or use a multi-session host) — NOT -a credential fix, a rerun, or a licensing change. - -Control-plane failure (pre-workflow), so there is no `process/` source; the -agent diagnoses from `uip or jobs`/`machines` evidence. Grading: -`skill_triggered` + `llm_judge` vs `RESOLUTION.md`. diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/d28a78f6ea.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/d28a78f6ea.json deleted file mode 100644 index dab6fc94b7..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/d28a78f6ea.json +++ /dev/null @@ -1,31 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1710001-1111-4111-8111-111111111111", - "StartTime": "2026-07-19T06:00:03.100Z", - "EndTime": "2026-07-19T06:00:03.900Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 50, - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Could not start executor. The workstation is in use by another user. Please retry after the user logs off or disconnects.\r\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\r\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\r\n", - "CreationTime": "2026-07-19T06:00:02.500Z", - "ReleaseName": "DailyExtract", - "Type": "Unattended", - "OutputArguments": "{}", - "HostMachineName": "WS-CLIENT-07", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "FolderKey": "f1a10001-0000-4000-8000-000000000001", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\SVC_EXTRACT", - "OrchestratorUserIdentity": "svc_extract", - "LastModificationTime": "2026-07-19T06:00:03.900Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a1710001111141118111111111111111", - "Id": 671330101 - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/dc064823f4.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/dc064823f4.json deleted file mode 100644 index 743c9aa9f3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/dc064823f4.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "MachineList", - "Data": [ - { - "name": "WS-CLIENT-07", - "description": "Windows 11 client workstation, dedicated unattended host.", - "type": "Template", - "key": "beef1001-2222-4333-8444-555566667701", - "id": 1990701 - } - ], - "Pagination": { - "Returned": 1, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/e0f3d81244.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/e0f3d81244.json deleted file mode 100644 index 7e2422a91a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/e0f3d81244.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "a1710001-1111-4111-8111-111111111111", - "State": "Faulted", - "ProcessName": "DailyExtract", - "ReleaseName": "DailyExtract", - "StartTime": "2026-07-19T06:00:03.100Z", - "EndTime": "2026-07-19T06:00:03.900Z", - "CreationTime": "2026-07-19T06:00:02.500Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/ee2628e6b6.json deleted file mode 100644 index e6011cd4c3..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Data": [ - { - "Key": "f1a10001-0000-4000-8000-000000000001", - "Name": "Unattended Ops", - "Path": "Unattended Ops", - "Description": "Scheduled unattended automations.", - "Type": "Standard", - "ParentKey": "" - }, - { - "Key": "c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/f5f2f217c8.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/f5f2f217c8.json deleted file mode 100644 index 7e2422a91a..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/f5f2f217c8.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "Result": "Success", - "Code": "JobsList", - "Data": [ - { - "Key": "a1710001-1111-4111-8111-111111111111", - "State": "Faulted", - "ProcessName": "DailyExtract", - "ReleaseName": "DailyExtract", - "StartTime": "2026-07-19T06:00:03.100Z", - "EndTime": "2026-07-19T06:00:03.900Z", - "CreationTime": "2026-07-19T06:00:02.500Z", - "HostMachineName": "WS-CLIENT-07", - "Type": "Unattended", - "Source": "Schedule", - "Priority": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/manifest.json deleted file mode 100644 index cc997f160e..0000000000 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/m/r/manifest.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the rpa-workstation-in-use scenario. The agent chains folders list -> jobs list --state Faulted -> jobs get -> jobs logs -> jobs history -> jobs list (concurrency) -> machines list --all-fields. Discriminating evidence: WS-CLIENT-07 is a Windows client (single interactive session) but its template grants unattendedSlots=3, and a concurrent job (HourlySync) was Running on the same host when DailyExtract faulted with 'workstation is in use by another user'. Specific rules precede generic ones (first-match-wins).", - "rules": [ - { "match": "or folders list", "file": "ee2628e6b6.json" }, - { "match": "or jobs list --folder-key f1a10001-0000-4000-8000-000000000001 --state Faulted", "file": "e0f3d81244.json" }, - { "match": "or jobs list --folder-key f1a10001-0000-4000-8000-000000000001", "file": "4b9e5c8cb4.json" }, - { "match": "or jobs list --state Faulted", "file": "f5f2f217c8.json" }, - { "match": "or jobs list", "file": "75949d659e.json" }, - { "match": "or jobs get a1710001-1111-4111-8111-111111111111", "file": "d28a78f6ea.json" }, - { "match": "or jobs logs a1710001-1111-4111-8111-111111111111 --level Error", "file": "356bfb7143.json" }, - { "match": "or jobs logs a1710001-1111-4111-8111-111111111111", "file": "7a6e65f01a.json" }, - { "match": "or jobs history a1710001-1111-4111-8111-111111111111", "file": "8b13f09c9d.json" }, - { "match": "or machines list --all-fields", "file": "8883fea2d6.json" }, - { "match": "or machines list", "file": "dc064823f4.json" }, - { "match": "docsai ask", "passthrough": true } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/uip-fixture.json new file mode 100644 index 0000000000..04e58d0fbc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/data/uip-fixture.json @@ -0,0 +1,134 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1a10001-0000-4000-8000-000000000001", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1710001-1111-4111-8111-111111111111\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyExtract\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"StartTime\": \"2026-07-19T06:00:03.100Z\",\r\n \"EndTime\": \"2026-07-19T06:00:03.900Z\",\r\n \"CreationTime\": \"2026-07-19T06:00:02.500Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1710001-1111-4111-8111-111111111111", + "--level", + "Error" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T06:00:03.820Z\",\r\n \"Message\": \"Could not start executor. The workstation is in use by another user. Please retry after the user logs off or disconnects.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--folder-key", + "f1a10001-0000-4000-8000-000000000001" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1710001-1111-4111-8111-111111111111\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyExtract\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"StartTime\": \"2026-07-19T06:00:03.100Z\",\r\n \"EndTime\": \"2026-07-19T06:00:03.900Z\",\r\n \"CreationTime\": \"2026-07-19T06:00:02.500Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"a1710002-2222-4222-8222-222222222222\",\r\n \"State\": \"Running\",\r\n \"ProcessName\": \"HourlySync\",\r\n \"ReleaseName\": \"HourlySync\",\r\n \"StartTime\": \"2026-07-19T06:00:01.400Z\",\r\n \"EndTime\": null,\r\n \"CreationTime\": \"2026-07-19T06:00:00.900Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"a1710003-3333-4333-8333-333333333333\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"DailyExtract\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"StartTime\": \"2026-07-18T06:00:02.700Z\",\r\n \"EndTime\": \"2026-07-18T06:04:55.100Z\",\r\n \"CreationTime\": \"2026-07-18T06:00:02.100Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list", + "--state", + "Faulted" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1710001-1111-4111-8111-111111111111\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyExtract\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"StartTime\": \"2026-07-19T06:00:03.100Z\",\r\n \"EndTime\": \"2026-07-19T06:00:03.900Z\",\r\n \"CreationTime\": \"2026-07-19T06:00:02.500Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a1710001-1111-4111-8111-111111111111" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1710001-1111-4111-8111-111111111111\",\r\n \"StartTime\": \"2026-07-19T06:00:03.100Z\",\r\n \"EndTime\": \"2026-07-19T06:00:03.900Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 50,\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Could not start executor. The workstation is in use by another user. Please retry after the user logs off or disconnects.\\r\\n at UiPath.Robot.Service.SessionManager.CreateSessionAsync(LogonRequest request)\\r\\n at UiPath.Robot.Service.JobManager.StartJobAsync(JobStartRequest request, CancellationToken cancellationToken)\\r\\n\",\r\n \"CreationTime\": \"2026-07-19T06:00:02.500Z\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"Type\": \"Unattended\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"FolderKey\": \"f1a10001-0000-4000-8000-000000000001\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\SVC_EXTRACT\",\r\n \"OrchestratorUserIdentity\": \"svc_extract\",\r\n \"LastModificationTime\": \"2026-07-19T06:00:03.900Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a1710001111141118111111111111111\",\r\n \"Id\": 671330101\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1710001-1111-4111-8111-111111111111" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T06:00:03.820Z\",\r\n \"Message\": \"Could not start executor. The workstation is in use by another user. Please retry after the user logs off or disconnects.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "a1710001-1111-4111-8111-111111111111" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-19T06:00:03.900Z\",\r\n \"JobId\": 671330101\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-19T06:00:03.110Z\",\r\n \"JobId\": 671330101\r\n },\r\n {\r\n \"State\": \"Pending\",\r\n \"Timestamp\": \"2026-07-19T06:00:02.540Z\",\r\n \"JobId\": 671330101\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list", + "--all-fields" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"licenseKey\": \"605b37db-dc58-460f-99ed-93fcfb2f2701\",\r\n \"name\": \"WS-CLIENT-07\",\r\n \"description\": \"Windows 11 client workstation, dedicated unattended host.\",\r\n \"type\": \"Template\",\r\n \"scope\": \"Default\",\r\n \"nonProductionSlots\": 0,\r\n \"unattendedSlots\": 3,\r\n \"headlessSlots\": 0,\r\n \"testAutomationSlots\": 0,\r\n \"hostingSlots\": 0,\r\n \"appTestSlots\": 0,\r\n \"performanceTestSlots\": 0,\r\n \"agentSlots\": 0,\r\n \"automationCloudSlots\": 0,\r\n \"automationCloudTestAutomationSlots\": 0,\r\n \"key\": \"beef1001-2222-4333-8444-555566667701\",\r\n \"endpointDetectionStatus\": \"Available\",\r\n \"robotVersions\": [\r\n {\r\n \"count\": 1,\r\n \"version\": \"24.10.4\"\r\n }\r\n ],\r\n \"robotUsers\": [\r\n {\r\n \"userName\": \"UIPATH\\\\SVC_EXTRACT\"\r\n }\r\n ],\r\n \"automationType\": \"Any\",\r\n \"targetFramework\": \"Windows\",\r\n \"tags\": [],\r\n \"id\": 1990701\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"f1a10001-0000-4000-8000-000000000001\",\r\n \"Name\": \"Unattended Ops\",\r\n \"Path\": \"Unattended Ops\",\r\n \"Description\": \"Scheduled unattended automations.\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"c1d2e3f4-a5b6-4c78-9d0e-1f2a3b4c5d6f\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobsList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"a1710001-1111-4111-8111-111111111111\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"DailyExtract\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"StartTime\": \"2026-07-19T06:00:03.100Z\",\r\n \"EndTime\": \"2026-07-19T06:00:03.900Z\",\r\n \"CreationTime\": \"2026-07-19T06:00:02.500Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"a1710002-2222-4222-8222-222222222222\",\r\n \"State\": \"Running\",\r\n \"ProcessName\": \"HourlySync\",\r\n \"ReleaseName\": \"HourlySync\",\r\n \"StartTime\": \"2026-07-19T06:00:01.400Z\",\r\n \"EndTime\": null,\r\n \"CreationTime\": \"2026-07-19T06:00:00.900Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n },\r\n {\r\n \"Key\": \"a1710003-3333-4333-8333-333333333333\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"DailyExtract\",\r\n \"ReleaseName\": \"DailyExtract\",\r\n \"StartTime\": \"2026-07-18T06:00:02.700Z\",\r\n \"EndTime\": \"2026-07-18T06:04:55.100Z\",\r\n \"CreationTime\": \"2026-07-18T06:00:02.100Z\",\r\n \"HostMachineName\": \"WS-CLIENT-07\",\r\n \"Type\": \"Unattended\",\r\n \"Source\": \"Schedule\",\r\n \"Priority\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "machines", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"MachineList\",\r\n \"Data\": [\r\n {\r\n \"name\": \"WS-CLIENT-07\",\r\n \"description\": \"Windows 11 client workstation, dedicated unattended host.\",\r\n \"type\": \"Template\",\r\n \"key\": \"beef1001-2222-4333-8444-555566667701\",\r\n \"id\": 1990701\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/task.yaml b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/task.yaml index 8c308d0d43..c45526b6f7 100644 --- a/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/task.yaml +++ b/tests/tasks/uipath-troubleshoot/products/orchestrator/rpa-workstation-in-use/task.yaml @@ -17,20 +17,12 @@ run_limits: sandbox: python: {} - template_sources: - - type: template_dir - path: ../../../_shared/mock_template - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/README.md new file mode 100644 index 0000000000..1eb4b4a665 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/README.md @@ -0,0 +1,27 @@ +# Argument Exception — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.ArgumentException` thrown from an `Assign` expression in +`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` +CLI mock and must reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Assign` **Parse Day Of Week** in `Main.xaml` runs `Enum.Parse(typeof(DayOfWeek), inputDay)` where `inputDay` is `"Funday"`. `"Funday"` is not a defined `DayOfWeek` name, so `Enum.Parse` throws `System.ArgumentException: Requested value 'Funday' was not found.` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/argument-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/134734f3a4.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/134734f3a4.json deleted file mode 100644 index 43926195fd..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/134734f3a4.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b2d3f4a5-2222-4abc-9def-000000000002", - "StartTime": "2026-06-23T13:02:40.000Z", - "EndTime": "2026-06-23T13:02:40.610Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b2d3f4a5-2222-4abc-9def-000000000002", - "Info": "Requested value 'Funday' was not found.\r\n\r\nin Main.xaml\r\n at Assign \"Parse Day Of Week\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentException: Requested value 'Funday' was not found.\r\n at System.Enum.Parse(Type enumType, String value, Boolean ignoreCase)\r\n at System.Enum.Parse(Type enumType, String value)\r\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_b2d3f4a5sp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-23T13:02:40.000Z", - "ReleaseName": "ScheduleParser", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10002, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-23T13:02:40.610Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b2d3f4a522224abc9def000000000002", - "Id": 4015002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/4bfc2a44bd.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/4bfc2a44bd.json deleted file mode 100644 index 34c8ed8718..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/4bfc2a44bd.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-23T13:02:40.330Z", - "Message": "Parse Day Of Week: Requested value 'Funday' was not found.\r\nin Main.xaml\r\n at Assign \"Parse Day Of Week\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentException\r\n at System.Enum.Parse(Type enumType, String value, Boolean ignoreCase)\r\n at System.Enum.Parse(Type enumType, String value)\r\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_b2d3f4a5sp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/README.md deleted file mode 100644 index 555aa863c4..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Argument Exception — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.ArgumentException` thrown from an `Assign` expression in -`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` -CLI mock and must reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Assign` **Parse Day Of Week** in `Main.xaml` runs `Enum.Parse(typeof(DayOfWeek), inputDay)` where `inputDay` is `"Funday"`. `"Funday"` is not a defined `DayOfWeek` name, so `Enum.Parse` throws `System.ArgumentException: Requested value 'Funday' was not found.` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/argument-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/manifest.json deleted file mode 100644 index 2808e8dde8..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get b2d3f4a5-2222-4abc-9def-000000000002 --output json", - "file": "134734f3a4.json" - }, - { - "match": "or jobs logs b2d3f4a5-2222-4abc-9def-000000000002 --level Error --output json", - "file": "4bfc2a44bd.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get b2d3f4a5-2222-4abc-9def-000000000002", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs b2d3f4a5-2222-4abc-9def-000000000002", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/uip-fixture.json new file mode 100644 index 0000000000..f797295bc9 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "b2d3f4a5-2222-4abc-9def-000000000002", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-23T13:02:40.330Z\",\r\n \"Message\": \"Parse Day Of Week: Requested value 'Funday' was not found.\\r\\nin Main.xaml\\r\\n at Assign \\\"Parse Day Of Week\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentException\\r\\n at System.Enum.Parse(Type enumType, String value, Boolean ignoreCase)\\r\\n at System.Enum.Parse(Type enumType, String value)\\r\\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_b2d3f4a5sp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b2d3f4a5-2222-4abc-9def-000000000002", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b2d3f4a5-2222-4abc-9def-000000000002\",\r\n \"StartTime\": \"2026-06-23T13:02:40.000Z\",\r\n \"EndTime\": \"2026-06-23T13:02:40.610Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b2d3f4a5-2222-4abc-9def-000000000002\",\r\n \"Info\": \"Requested value 'Funday' was not found.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Parse Day Of Week\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentException: Requested value 'Funday' was not found.\\r\\n at System.Enum.Parse(Type enumType, String value, Boolean ignoreCase)\\r\\n at System.Enum.Parse(Type enumType, String value)\\r\\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_b2d3f4a5sp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_b2d3f4a5sp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-23T13:02:40.000Z\",\r\n \"ReleaseName\": \"ScheduleParser\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10002,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-23T13:02:40.610Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b2d3f4a522224abc9def000000000002\",\r\n \"Id\": 4015002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get b2d3f4a5-2222-4abc-9def-000000000002", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs b2d3f4a5-2222-4abc-9def-000000000002", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/task.yaml index f98682f855..0baf6b58c3 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/README.md new file mode 100644 index 0000000000..3e9e8fd0b8 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/README.md @@ -0,0 +1,36 @@ +# Argument Null Exception — Faithful Replay + +This scenario replays a real UiPath diagnostic investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +Root Cause: An explicit `null` literal is assigned to the variable bound to the Copy File activity's Path (and Destination) in Main.xaml, so the activity passes null to `new DirectoryInfo(...)`, which throws `System.ArgumentNullException`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-diagnostics` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name argument-null-exception --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/2ccc0db107.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/2ccc0db107.json deleted file mode 100644 index 4e4e9fe7d1..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/2ccc0db107.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Info", - "TimeStamp": "2026-05-13T08:42:12.922Z", - "Message": "ERN execution ended" - }, - { - "Level": "Error", - "TimeStamp": "2026-05-13T08:42:11.563Z", - "Message": "Copy File: Value cannot be null. (Parameter 'path')\r\nin Main.xaml\r\n at CopyFile \"Copy File\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentNullException\r\n at System.ArgumentNullException.Throw(String paramName)\r\n at System.IO.DirectoryInfo..ctor(String path)\r\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - }, - { - "Level": "Info", - "TimeStamp": "2026-05-13T08:42:11.520Z", - "Message": "ERN execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/5f86257b7b.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/5f86257b7b.json deleted file mode 100644 index cc7f032c5b..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/5f86257b7b.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "ProcessList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "B16092A1-0C68-436C-BF3A-147AB53B6762", - "Name": "ERN", - "ProcessKey": "ERN", - "ProcessVersion": "1.0.6", - "Description": "Blank Process", - "IsLatestVersion": true, - "RetentionPeriod": 30, - "RetentionAction": "Delete", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "FolderPath": "Shared" - }, - { - "Key": "EEAD4A4E-A3D9-44E5-A8EA-644B5B4BC2B6", - "Name": "BlankProcess", - "ProcessKey": "BlankProcess", - "ProcessVersion": "1.0.3", - "Description": "Blank Process", - "IsLatestVersion": true, - "RetentionPeriod": 30, - "RetentionAction": "Delete", - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "FolderPath": "Shared" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/9c6c160a6b.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/9c6c160a6b.json deleted file mode 100644 index 0c91605aff..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/9c6c160a6b.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-05-13T08:42:12.997Z", - "JobId": 4014089 - }, - { - "State": "Running", - "Timestamp": "2026-05-13T08:42:11.210Z", - "JobId": 4014089 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/README.md deleted file mode 100644 index 3ffdc2632b..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Argument Null Exception — Faithful Replay - -This scenario replays a real UiPath diagnostic investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -Root Cause: An explicit `null` literal is assigned to the variable bound to the Copy File activity's Path (and Destination) in Main.xaml, so the activity passes null to `new DirectoryInfo(...)`, which throws `System.ArgumentNullException`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-diagnostics` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-diagnostics/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name argument-null-exception --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/bcfa6ee3d9.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/bcfa6ee3d9.json deleted file mode 100644 index ab8c89ca80..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/bcfa6ee3d9.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [ - { - "PermissionStatus": 0, - "Id": "00000000-0000-0000-b98b-59af3439c595", - "TraceId": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "ParentId": null, - "Name": "Robot Job 0686cab1-35c6-4d92-aed2-c91d9c275524", - "StartTime": "2026-05-13T08:42:11.4796773Z", - "EndTime": "2026-05-13T08:42:11.5741896Z", - "Attributes": "{\"Exception\":\"System.ArgumentNullException: Value cannot be null. (Parameter 'path')\\r\\n at System.ArgumentNullException.Throw(String paramName)\\r\\n at System.IO.DirectoryInfo..ctor(String path)\\r\\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"}", - "Status": 2, - "OrganizationId": "1215ce35-89a9-4dff-9f40-19c57da7689e", - "TenantId": "bcae8058-3f3b-4ec2-9550-517007673698", - "ExpiryTimeUtc": null, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Source": 4, - "SpanType": "RobotJob", - "ProcessKey": "b16092a1-0c68-436c-bf3a-147ab53b6762", - "JobKey": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "ReferenceId": null, - "VerbosityLevel": 2, - "ExecutionType": null, - "UpdatedAt": "2026-05-13T08:42:12.8439788Z", - "AgentVersion": null, - "Context": null, - "Attachments": null - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/d26f2ef0c0.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/d26f2ef0c0.json deleted file mode 100644 index 1bfe2a45e3..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/d26f2ef0c0.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "StartTime": "2026-05-13T08:42:11.170Z", - "EndTime": "2026-05-13T08:42:12.977Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Agent", - "SourceType": "Agent", - "BatchExecutionKey": "0686cab1-35c6-4d92-aed2-c91d9c275524", - "Info": "Value cannot be null. (Parameter 'path')\r\n\r\nin Main.xaml\r\n at CopyFile \"Copy File\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'path')\r\n at System.ArgumentNullException.Throw(String paramName)\r\n at System.IO.DirectoryInfo..ctor(String path)\r\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-05-13T08:42:11.170Z", - "ReleaseName": "ERN", - "Type": "Attended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "StudioPro", - "RequiresUserInteraction": true, - "ReleaseVersionId": 53410, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\REPLACEMENT_USER", - "OrchestratorUserIdentity": "original_email@test.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-05-13T08:42:12.977Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "0686cab135c64d92aed2c91d9c275524", - "AutopilotForRobots": { - "Enabled": true, - "HealingEnabled": true - }, - "Id": 4014089 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/f3e78647da.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/f3e78647da.json deleted file mode 100644 index deda9af774..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/f3e78647da.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T08:42:11.563Z", - "Message": "Copy File: Value cannot be null. (Parameter 'path')\r\nin Main.xaml\r\n at CopyFile \"Copy File\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentNullException\r\n at System.ArgumentNullException.Throw(String paramName)\r\n at System.IO.DirectoryInfo..ctor(String path)\r\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/manifest.json deleted file mode 100644 index 005298796f..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/m/r/manifest.json +++ /dev/null @@ -1,71 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py - review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 0686cab1-35c6-4d92-aed2-c91d9c275524 --output json", - "file": "d26f2ef0c0.json" - }, - { - "match": "or jobs logs 0686cab1-35c6-4d92-aed2-c91d9c275524 --level Error --output json", - "file": "f3e78647da.json" - }, - { - "match": "or jobs history 0686cab1-35c6-4d92-aed2-c91d9c275524 --output json", - "file": "9c6c160a6b.json" - }, - { - "match": "or jobs logs 0686cab1-35c6-4d92-aed2-c91d9c275524 --output json", - "file": "2ccc0db107.json" - }, - { - "match": "or jobs traces 0686cab1-35c6-4d92-aed2-c91d9c275524 --output json", - "file": "bcfa6ee3d9.json" - }, - { - "match": "or processes list --folder-key 1965a46b-db4e-469e-aaaa-7e0b379cb34d --output json", - "file": "5f86257b7b.json" - }, - { - "match": "docsai ask", - "passthrough": true, - "_doc": "Per CLAUDE.md: docsai queries always passthrough - query strings vary between runs and canned responses go stale. Cached per-query within a run." - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or jobs get 0686cab1-35c6-4d92-aed2-c91d9c275524", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs 0686cab1-35c6-4d92-aed2-c91d9c275524", - "min": 1, - "description": "Agent reads error-level job logs for additional execution detail." - }, - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or processes list", - "min": 0, - "description": "OPTIONAL - agent may inspect the process/release configuration." - }, - { - "pattern": "or jobs traces 0686cab1-35c6-4d92-aed2-c91d9c275524", - "min": 0, - "description": "OPTIONAL - agent may pull execution traces for additional detail." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/uip-fixture.json new file mode 100644 index 0000000000..411e9ba780 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/data/uip-fixture.json @@ -0,0 +1,129 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "0686cab1-35c6-4d92-aed2-c91d9c275524", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T08:42:11.563Z\",\r\n \"Message\": \"Copy File: Value cannot be null. (Parameter 'path')\\r\\nin Main.xaml\\r\\n at CopyFile \\\"Copy File\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at System.ArgumentNullException.Throw(String paramName)\\r\\n at System.IO.DirectoryInfo..ctor(String path)\\r\\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "processes", + "list", + "--folder-key", + "1965a46b-db4e-469e-aaaa-7e0b379cb34d", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"ProcessList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"B16092A1-0C68-436C-BF3A-147AB53B6762\",\r\n \"Name\": \"ERN\",\r\n \"ProcessKey\": \"ERN\",\r\n \"ProcessVersion\": \"1.0.6\",\r\n \"Description\": \"Blank Process\",\r\n \"IsLatestVersion\": true,\r\n \"RetentionPeriod\": 30,\r\n \"RetentionAction\": \"Delete\",\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"FolderPath\": \"Shared\"\r\n },\r\n {\r\n \"Key\": \"EEAD4A4E-A3D9-44E5-A8EA-644B5B4BC2B6\",\r\n \"Name\": \"BlankProcess\",\r\n \"ProcessKey\": \"BlankProcess\",\r\n \"ProcessVersion\": \"1.0.3\",\r\n \"Description\": \"Blank Process\",\r\n \"IsLatestVersion\": true,\r\n \"RetentionPeriod\": 30,\r\n \"RetentionAction\": \"Delete\",\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"FolderPath\": \"Shared\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "0686cab1-35c6-4d92-aed2-c91d9c275524", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"StartTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"EndTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Agent\",\r\n \"SourceType\": \"Agent\",\r\n \"BatchExecutionKey\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"Info\": \"Value cannot be null. (Parameter 'path')\\r\\n\\r\\nin Main.xaml\\r\\n at CopyFile \\\"Copy File\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentNullException: Value cannot be null. (Parameter 'path')\\r\\n at System.ArgumentNullException.Throw(String paramName)\\r\\n at System.IO.DirectoryInfo..ctor(String path)\\r\\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-05-13T08:42:11.170Z\",\r\n \"ReleaseName\": \"ERN\",\r\n \"Type\": \"Attended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"StudioPro\",\r\n \"RequiresUserInteraction\": true,\r\n \"ReleaseVersionId\": 53410,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\REPLACEMENT_USER\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-05-13T08:42:12.977Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"0686cab135c64d92aed2c91d9c275524\",\r\n \"AutopilotForRobots\": {\r\n \"Enabled\": true,\r\n \"HealingEnabled\": true\r\n },\r\n \"Id\": 4014089\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history", + "0686cab1-35c6-4d92-aed2-c91d9c275524", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-05-13T08:42:12.997Z\",\r\n \"JobId\": 4014089\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-05-13T08:42:11.210Z\",\r\n \"JobId\": 4014089\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "0686cab1-35c6-4d92-aed2-c91d9c275524", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T08:42:12.922Z\",\r\n \"Message\": \"ERN execution ended\"\r\n },\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T08:42:11.563Z\",\r\n \"Message\": \"Copy File: Value cannot be null. (Parameter 'path')\\r\\nin Main.xaml\\r\\n at CopyFile \\\"Copy File\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentNullException\\r\\n at System.ArgumentNullException.Throw(String paramName)\\r\\n at System.IO.DirectoryInfo..ctor(String path)\\r\\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n },\r\n {\r\n \"Level\": \"Info\",\r\n \"TimeStamp\": \"2026-05-13T08:42:11.520Z\",\r\n \"Message\": \"ERN execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces", + "0686cab1-35c6-4d92-aed2-c91d9c275524", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": [\r\n {\r\n \"PermissionStatus\": 0,\r\n \"Id\": \"00000000-0000-0000-b98b-59af3439c595\",\r\n \"TraceId\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"ParentId\": null,\r\n \"Name\": \"Robot Job 0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"StartTime\": \"2026-05-13T08:42:11.4796773Z\",\r\n \"EndTime\": \"2026-05-13T08:42:11.5741896Z\",\r\n \"Attributes\": \"{\\\"Exception\\\":\\\"System.ArgumentNullException: Value cannot be null. (Parameter 'path')\\\\r\\\\n at System.ArgumentNullException.Throw(String paramName)\\\\r\\\\n at System.IO.DirectoryInfo..ctor(String path)\\\\r\\\\n at UiPath.Core.Activities.CopyFile.Execute(CodeActivityContext context)\\\\r\\\\n at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\\\r\\\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\\\"}\",\r\n \"Status\": 2,\r\n \"OrganizationId\": \"1215ce35-89a9-4dff-9f40-19c57da7689e\",\r\n \"TenantId\": \"bcae8058-3f3b-4ec2-9550-517007673698\",\r\n \"ExpiryTimeUtc\": null,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Source\": 4,\r\n \"SpanType\": \"RobotJob\",\r\n \"ProcessKey\": \"b16092a1-0c68-436c-bf3a-147ab53b6762\",\r\n \"JobKey\": \"0686cab1-35c6-4d92-aed2-c91d9c275524\",\r\n \"ReferenceId\": null,\r\n \"VerbosityLevel\": 2,\r\n \"ExecutionType\": null,\r\n \"UpdatedAt\": \"2026-05-13T08:42:12.8439788Z\",\r\n \"AgentVersion\": null,\r\n \"Context\": null,\r\n \"Attachments\": null\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or jobs get 0686cab1-35c6-4d92-aed2-c91d9c275524", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs 0686cab1-35c6-4d92-aed2-c91d9c275524", + "min": 1, + "description": "Agent reads error-level job logs for additional execution detail." + }, + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or processes list", + "min": 0, + "description": "OPTIONAL - agent may inspect the process/release configuration." + }, + { + "pattern": "or jobs traces 0686cab1-35c6-4d92-aed2-c91d9c275524", + "min": 0, + "description": "OPTIONAL - agent may pull execution traces for additional detail." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/task.yaml index 4ad5a0f760..d2e4d94f06 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-null-exception/task.yaml @@ -18,22 +18,14 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/README.md new file mode 100644 index 0000000000..f9b8427be2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/README.md @@ -0,0 +1,27 @@ +# Argument Out Of Range Exception — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.ArgumentOutOfRangeException` thrown from an `Assign` expression in +`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` +CLI mock and must reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Assign` **Slice Year From Invoice Id** in `Main.xaml` runs `invoiceId.Substring(10)`, but `invoiceId` is `"INV-7"` (length 5). A start index of 10 is past the end of the string, so `String.Substring` throws `System.ArgumentOutOfRangeException: startIndex cannot be larger than length of string. (Parameter 'startIndex')`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/argument-out-of-range-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/6a1ed47cf9.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/6a1ed47cf9.json deleted file mode 100644 index 5e1aa747d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/6a1ed47cf9.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-27T15:05:33.190Z", - "Message": "Slice Year From Invoice Id: startIndex cannot be larger than length of string. (Parameter 'startIndex')\r\nin Main.xaml\r\n at Assign \"Slice Year From Invoice Id\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.ArgumentOutOfRangeException\r\n at System.String.Substring(Int32 startIndex)\r\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_f6b7d8e9is.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/README.md deleted file mode 100644 index 3cdf5ec292..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Argument Out Of Range Exception — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.ArgumentOutOfRangeException` thrown from an `Assign` expression in -`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` -CLI mock and must reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Assign` **Slice Year From Invoice Id** in `Main.xaml` runs `invoiceId.Substring(10)`, but `invoiceId` is `"INV-7"` (length 5). A start index of 10 is past the end of the string, so `String.Substring` throws `System.ArgumentOutOfRangeException: startIndex cannot be larger than length of string. (Parameter 'startIndex')`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/argument-out-of-range-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/a85c1ab899.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/a85c1ab899.json deleted file mode 100644 index 2c9f5b7578..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/a85c1ab899.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "f6b7d8e9-6666-4abc-9def-000000000006", - "StartTime": "2026-06-27T15:05:33.000Z", - "EndTime": "2026-06-27T15:05:33.420Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "f6b7d8e9-6666-4abc-9def-000000000006", - "Info": "startIndex cannot be larger than length of string. (Parameter 'startIndex')\r\n\r\nin Main.xaml\r\n at Assign \"Slice Year From Invoice Id\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.ArgumentOutOfRangeException: startIndex cannot be larger than length of string. (Parameter 'startIndex')\r\n at System.String.Substring(Int32 startIndex)\r\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_f6b7d8e9is.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-27T15:05:33.000Z", - "ReleaseName": "InvoiceIdSlicer", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10006, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-27T15:05:33.420Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "f6b7d8e966664abc9def000000000006", - "Id": 4015006 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/manifest.json deleted file mode 100644 index 82bc651409..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get f6b7d8e9-6666-4abc-9def-000000000006 --output json", - "file": "a85c1ab899.json" - }, - { - "match": "or jobs logs f6b7d8e9-6666-4abc-9def-000000000006 --level Error --output json", - "file": "6a1ed47cf9.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get f6b7d8e9-6666-4abc-9def-000000000006", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs f6b7d8e9-6666-4abc-9def-000000000006", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/uip-fixture.json new file mode 100644 index 0000000000..4024f6da66 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "f6b7d8e9-6666-4abc-9def-000000000006", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-27T15:05:33.190Z\",\r\n \"Message\": \"Slice Year From Invoice Id: startIndex cannot be larger than length of string. (Parameter 'startIndex')\\r\\nin Main.xaml\\r\\n at Assign \\\"Slice Year From Invoice Id\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.ArgumentOutOfRangeException\\r\\n at System.String.Substring(Int32 startIndex)\\r\\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_f6b7d8e9is.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "f6b7d8e9-6666-4abc-9def-000000000006", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"f6b7d8e9-6666-4abc-9def-000000000006\",\r\n \"StartTime\": \"2026-06-27T15:05:33.000Z\",\r\n \"EndTime\": \"2026-06-27T15:05:33.420Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"f6b7d8e9-6666-4abc-9def-000000000006\",\r\n \"Info\": \"startIndex cannot be larger than length of string. (Parameter 'startIndex')\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Slice Year From Invoice Id\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.ArgumentOutOfRangeException: startIndex cannot be larger than length of string. (Parameter 'startIndex')\\r\\n at System.String.Substring(Int32 startIndex)\\r\\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_f6b7d8e9is.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_f6b7d8e9is.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-27T15:05:33.000Z\",\r\n \"ReleaseName\": \"InvoiceIdSlicer\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10006,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-27T15:05:33.420Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"f6b7d8e966664abc9def000000000006\",\r\n \"Id\": 4015006\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get f6b7d8e9-6666-4abc-9def-000000000006", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs f6b7d8e9-6666-4abc-9def-000000000006", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/task.yaml index 316b45bf01..e6a2f7c7df 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/argument-out-of-range-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/README.md new file mode 100644 index 0000000000..cebf1757ed --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/README.md @@ -0,0 +1,32 @@ +# Assign — "The source contains no data rows" (LINQ / CopyToDataTable, Runtime) + +Runtime troubleshooting scenario for `System.Activities.Statements.Assign`. + +## What this scenario exercises + +An unattended job faults with `System.InvalidOperationException: The source contains no data rows`. The +agent must trace it to an `Assign` running a LINQ `.Where(...).CopyToDataTable()` whose filter matched +**zero rows** on this run, and recognize that `CopyToDataTable()` throws on an empty sequence. The fix +is to guard the materialization with `.Any()` (assign only when matches exist; otherwise `.Clone()` / +handle the empty case). The source table is non-null and populated (42 rows loaded) — this is NOT a +`NullReferenceException`. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted VB project source: `Main.xaml` with an `Assign` "Filter active invoices" whose `Value` is `dtInvoices.AsEnumerable().Where(...).CopyToDataTable()` with no `.Any()` guard | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the unguarded `CopyToDataTable()` on a zero-match LINQ filter and the fix (guard with + `.Any()` / handle the empty result), and did NOT misdiagnose it as a null/uninitialized-table error. + +Playbook: `references/runtime-exceptions/playbooks/assign-linq-no-data-rows.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/3db6b96527.json deleted file mode 100644 index 49db49bbe8..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/3db6b96527.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b8f3c1d2-4a67-4e90-9c15-2d7e6f0a3b58", - "StartTime": "2026-07-19T03:05:11.240Z", - "EndTime": "2026-07-19T03:05:12.980Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Filter active invoices: The source contains no data rows.\r\n\r\nin Main.xaml\r\n at Assign \"Filter active invoices\"\r\n at Sequence \"Main Sequence\"\r\n at DynamicActivity \"Main\"\r\n\r\nSystem.InvalidOperationException: The source contains no data rows.\r\n at System.Data.DataTableExtensions.CopyToDataTable[T](IEnumerable`1 source)\r\n at InvoiceReconciliation_Expressions.__Expr1Get(Object[] locations, IEnumerable`1 executionProperties)\r\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\r\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\r\n at System.Activities.InArgument`1.TryPopulateValue(...)", - "CreationTime": "2026-07-19T03:05:11.240Z", - "ReleaseName": "InvoiceReconciliation", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-19T03:05:12.980Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b8f3c1d24a674e909c152d7e6f0a3b58", - "Id": 66812345 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/701433db1c.json deleted file mode 100644 index e553f82fde..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/701433db1c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Faulted", - "Timestamp": "2026-07-19T03:05:12.980Z", - "JobId": 66812345 - }, - { - "State": "Running", - "Timestamp": "2026-07-19T03:05:11.680Z", - "JobId": 66812345 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/75949d659e.json deleted file mode 100644 index 7857def1f4..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/75949d659e.json +++ /dev/null @@ -1,48 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "b8f3c1d2-4a67-4e90-9c15-2d7e6f0a3b58", - "State": "Faulted", - "ProcessName": "InvoiceReconciliation", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-19T03:05:11.240Z", - "EndTime": "2026-07-19T03:05:12.980Z", - "CreationTime": "2026-07-19T03:05:11.240Z" - }, - { - "Key": "1c7a9e04-5d38-4b21-8f60-3a2c7e1b9d45", - "State": "Successful", - "ProcessName": "InvoiceReconciliation", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-18T03:05:10.910Z", - "EndTime": "2026-07-18T03:06:02.330Z", - "CreationTime": "2026-07-18T03:05:10.910Z" - }, - { - "Key": "9e2b0d73-6a41-4c58-9032-4d8c1a5f7b26", - "State": "Successful", - "ProcessName": "InvoiceReconciliation", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-17T03:05:11.520Z", - "EndTime": "2026-07-17T03:06:05.870Z", - "CreationTime": "2026-07-17T03:05:11.520Z" - } - ], - "Pagination": { - "Returned": 3, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/README.md deleted file mode 100644 index 1bec42b8c1..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Assign — "The source contains no data rows" (LINQ / CopyToDataTable, Runtime) - -Runtime troubleshooting scenario for `System.Activities.Statements.Assign`. - -## What this scenario exercises - -An unattended job faults with `System.InvalidOperationException: The source contains no data rows`. The -agent must trace it to an `Assign` running a LINQ `.Where(...).CopyToDataTable()` whose filter matched -**zero rows** on this run, and recognize that `CopyToDataTable()` throws on an empty sequence. The fix -is to guard the materialization with `.Any()` (assign only when matches exist; otherwise `.Clone()` / -handle the empty case). The source table is non-null and populated (42 rows loaded) — this is NOT a -`NullReferenceException`. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted VB project source: `Main.xaml` with an `Assign` "Filter active invoices" whose `Value` is `dtInvoices.AsEnumerable().Where(...).CopyToDataTable()` with no `.Any()` guard | -| `data/m/r/*.json` | faulted-job fixtures — `job-get`/`job-logs` carry `The source contains no data rows` + a stack bottoming out in `CopyToDataTable` (user expression, not a package); `jobs-list` shows prior Successful runs (data-dependent); log shows "rows loaded: 42" (source non-empty) | -| `data/m/r/manifest.json` | `docsai ask` passthrough + `or jobs/folders` fixtures + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the unguarded `CopyToDataTable()` on a zero-match LINQ filter and the fix (guard with - `.Any()` / handle the empty result), and did NOT misdiagnose it as a null/uninitialized-table error. - -Playbook: `references/runtime-exceptions/playbooks/assign-linq-no-data-rows.md`. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/b49ae1b1b0.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/b49ae1b1b0.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/b49ae1b1b0.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/b8d18266dc.json deleted file mode 100644 index ce7c4abf6a..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-07-19T03:05:12.930Z", - "Message": "Filter active invoices: The source contains no data rows.\r\nin Main.xaml\r\n at Assign \"Filter active invoices\"\r\n at Sequence \"Main Sequence\"\r\n at DynamicActivity \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.InvalidOperationException: The source contains no data rows.\r\n at System.Data.DataTableExtensions.CopyToDataTable[T](IEnumerable`1 source)\r\n at InvoiceReconciliation_Expressions.__Expr1Get(Object[] locations, IEnumerable`1 executionProperties)\r\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-19T03:05:11.610Z", - "Message": "Invoice reconciliation started, rows loaded: 42" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-19T03:05:11.480Z", - "Message": "InvoiceReconciliation execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/ee2628e6b6.json deleted file mode 100644 index f71a080fc8..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/manifest.json deleted file mode 100644 index 61ad558734..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned faulted-job fixtures; any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "or jobs traces", - "file": "b49ae1b1b0.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/uip-fixture.json new file mode 100644 index 0000000000..42d2e3463f --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"b8f3c1d2-4a67-4e90-9c15-2d7e6f0a3b58\",\r\n \"State\": \"Faulted\",\r\n \"ProcessName\": \"InvoiceReconciliation\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-19T03:05:11.240Z\",\r\n \"EndTime\": \"2026-07-19T03:05:12.980Z\",\r\n \"CreationTime\": \"2026-07-19T03:05:11.240Z\"\r\n },\r\n {\r\n \"Key\": \"1c7a9e04-5d38-4b21-8f60-3a2c7e1b9d45\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"InvoiceReconciliation\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-18T03:05:10.910Z\",\r\n \"EndTime\": \"2026-07-18T03:06:02.330Z\",\r\n \"CreationTime\": \"2026-07-18T03:05:10.910Z\"\r\n },\r\n {\r\n \"Key\": \"9e2b0d73-6a41-4c58-9032-4d8c1a5f7b26\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"InvoiceReconciliation\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-17T03:05:11.520Z\",\r\n \"EndTime\": \"2026-07-17T03:06:05.870Z\",\r\n \"CreationTime\": \"2026-07-17T03:05:11.520Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 3,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-07-19T03:05:12.930Z\",\r\n \"Message\": \"Filter active invoices: The source contains no data rows.\\r\\nin Main.xaml\\r\\n at Assign \\\"Filter active invoices\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.InvalidOperationException: The source contains no data rows.\\r\\n at System.Data.DataTableExtensions.CopyToDataTable[T](IEnumerable`1 source)\\r\\n at InvoiceReconciliation_Expressions.__Expr1Get(Object[] locations, IEnumerable`1 executionProperties)\\r\\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T03:05:11.610Z\",\r\n \"Message\": \"Invoice reconciliation started, rows loaded: 42\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T03:05:11.480Z\",\r\n \"Message\": \"InvoiceReconciliation execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Faulted\",\r\n \"Timestamp\": \"2026-07-19T03:05:12.980Z\",\r\n \"JobId\": 66812345\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-19T03:05:11.680Z\",\r\n \"JobId\": 66812345\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b8f3c1d2-4a67-4e90-9c15-2d7e6f0a3b58\",\r\n \"StartTime\": \"2026-07-19T03:05:11.240Z\",\r\n \"EndTime\": \"2026-07-19T03:05:12.980Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Filter active invoices: The source contains no data rows.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Filter active invoices\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at DynamicActivity \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidOperationException: The source contains no data rows.\\r\\n at System.Data.DataTableExtensions.CopyToDataTable[T](IEnumerable`1 source)\\r\\n at InvoiceReconciliation_Expressions.__Expr1Get(Object[] locations, IEnumerable`1 executionProperties)\\r\\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\\r\\n at System.Activities.CodeActivity`1.InternalExecuteInResolutionContext(CodeActivityContext context)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(...)\",\r\n \"CreationTime\": \"2026-07-19T03:05:11.240Z\",\r\n \"ReleaseName\": \"InvoiceReconciliation\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-19T03:05:12.980Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b8f3c1d24a674e909c152d7e6f0a3b58\",\r\n \"Id\": 66812345\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/task.yaml index b6cc874fd9..5c226107e5 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-linq-no-data-rows/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/README.md new file mode 100644 index 0000000000..36c6d37c7d --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/README.md @@ -0,0 +1,33 @@ +# Assign — Type Mismatch (Object → String, Design-Time) + +Design-time (Studio) troubleshooting scenario for `System.Activities.Statements.Assign`. + +## What this scenario exercises + +Studio's Error List flags an `Assign` with `Cannot assign from type 'System.Object' to +'System.String'`. The agent must recognize this as a **type mismatch between the right-hand-side +expression and the target variable** — the RHS (`config("AccountName")` on a +`Dictionary(Of String, Object)`) returns `System.Object`, the target (`accountName`) is +`System.String` — and prescribe converting the RHS (`.ToString()` / cast) or realigning the target +type. This is a design-time validation error, not a robot job fault and not corrupt XAML. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted VB project source: `Main.xaml` with an `Assign` whose `To` is a `String` variable and whose `Value` is `config("AccountName")` where `config` is `Dictionary(Of String, Object)`; `project.json` sets `expressionLanguage: VisualBasic` | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the `Object`→`String` type mismatch on the `Assign` and the fix (convert the RHS + with `.ToString()` / an explicit cast, or change the target variable's type — do NOT rebuild the + `.xaml` or hunt for a faulted job). + +Playbook: `references/runtime-exceptions/playbooks/assign-type-mismatch.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/m/r/README.md deleted file mode 100644 index 653b8cefb9..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/m/r/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Assign — Type Mismatch (Object → String, Design-Time) - -Design-time (Studio) troubleshooting scenario for `System.Activities.Statements.Assign`. - -## What this scenario exercises - -Studio's Error List flags an `Assign` with `Cannot assign from type 'System.Object' to -'System.String'`. The agent must recognize this as a **type mismatch between the right-hand-side -expression and the target variable** — the RHS (`config("AccountName")` on a -`Dictionary(Of String, Object)`) returns `System.Object`, the target (`accountName`) is -`System.String` — and prescribe converting the RHS (`.ToString()` / cast) or realigning the target -type. This is a design-time validation error, not a robot job fault and not corrupt XAML. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted VB project source: `Main.xaml` with an `Assign` whose `To` is a `String` variable and whose `Value` is `config("AccountName")` where `config` is `Dictionary(Of String, Object)`; `project.json` sets `expressionLanguage: VisualBasic` | -| `data/m/r/manifest.json` | `docsai ask` passthrough + permissive empty `unmocked_default` — no job/trace exists for a design-time error | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the `Object`→`String` type mismatch on the `Assign` and the fix (convert the RHS - with `.ToString()` / an explicit cast, or change the target variable's type — do NOT rebuild the - `.xaml` or hunt for a faulted job). - -Playbook: `references/runtime-exceptions/playbooks/assign-type-mismatch.md`. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/m/r/manifest.json deleted file mode 100644 index 4626745300..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/m/r/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. Only docsai is proxied to the real CLI; any other uip call returns an empty array. This is a design-time Studio validation error with no Orchestrator job/log/trace, so no job fixtures exist. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/uip-fixture.json new file mode 100644 index 0000000000..bcb5501faa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/data/uip-fixture.json @@ -0,0 +1,8 @@ +{ + "version": 1, + "responses": [], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/task.yaml index e6e1ed60fc..6fef28df34 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/assign-type-mismatch-object-string/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/README.md new file mode 100644 index 0000000000..1c33500ede --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/README.md @@ -0,0 +1,27 @@ +# Directory Not Found Exception — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.IO.DirectoryNotFoundException` thrown from an `Assign` expression in +`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` +CLI mock and must reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Assign` **List Inbound Files** in `Main.xaml` runs `Directory.GetFiles(folderPath)` with `folderPath = "D:\Reports\2026\Inbound"`. That directory does not exist on the unattended robot host (host `MOCK-HOST`), so `Directory.GetFiles` throws `System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\Reports\2026\Inbound'.` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/directory-not-found-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/8f1041c953.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/8f1041c953.json deleted file mode 100644 index bf8deb9729..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/8f1041c953.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T06:45:02.210Z", - "Message": "List Inbound Files: Could not find a part of the path 'D:\\Reports\\2026\\Inbound'.\r\nin Main.xaml\r\n at Assign \"List Inbound Files\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IO.DirectoryNotFoundException\r\n at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)\r\n at System.IO.Enumeration.FileSystemEnumerator`1..ctor(String directory, Boolean isNormalized, EnumerationOptions options)\r\n at System.IO.Directory.GetFiles(String path)\r\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_c3e4a5b6fs.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/9eafe5f82c.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/9eafe5f82c.json deleted file mode 100644 index 7911d46a33..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/9eafe5f82c.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c3e4a5b6-3333-4abc-9def-000000000003", - "StartTime": "2026-06-24T06:45:02.000Z", - "EndTime": "2026-06-24T06:45:02.470Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "c3e4a5b6-3333-4abc-9def-000000000003", - "Info": "Could not find a part of the path 'D:\\Reports\\2026\\Inbound'.\r\n\r\nin Main.xaml\r\n at Assign \"List Inbound Files\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\\Reports\\2026\\Inbound'.\r\n at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)\r\n at System.IO.Enumeration.FileSystemEnumerator`1..ctor(String directory, Boolean isNormalized, EnumerationOptions options)\r\n at System.IO.Directory.GetFiles(String path)\r\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_c3e4a5b6fs.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-24T06:45:02.000Z", - "ReleaseName": "InboundFileSweep", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10003, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-24T06:45:02.470Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "c3e4a5b633334abc9def000000000003", - "Id": 4015003 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/README.md deleted file mode 100644 index e1226b7ed8..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Directory Not Found Exception — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.IO.DirectoryNotFoundException` thrown from an `Assign` expression in -`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` -CLI mock and must reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Assign` **List Inbound Files** in `Main.xaml` runs `Directory.GetFiles(folderPath)` with `folderPath = "D:\Reports\2026\Inbound"`. That directory does not exist on the unattended robot host (host `MOCK-HOST`), so `Directory.GetFiles` throws `System.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\Reports\2026\Inbound'.` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/directory-not-found-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/manifest.json deleted file mode 100644 index 21ad2d0874..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get c3e4a5b6-3333-4abc-9def-000000000003 --output json", - "file": "9eafe5f82c.json" - }, - { - "match": "or jobs logs c3e4a5b6-3333-4abc-9def-000000000003 --level Error --output json", - "file": "8f1041c953.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get c3e4a5b6-3333-4abc-9def-000000000003", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs c3e4a5b6-3333-4abc-9def-000000000003", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/uip-fixture.json new file mode 100644 index 0000000000..5a062c5c26 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "c3e4a5b6-3333-4abc-9def-000000000003", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T06:45:02.210Z\",\r\n \"Message\": \"List Inbound Files: Could not find a part of the path 'D:\\\\Reports\\\\2026\\\\Inbound'.\\r\\nin Main.xaml\\r\\n at Assign \\\"List Inbound Files\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IO.DirectoryNotFoundException\\r\\n at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)\\r\\n at System.IO.Enumeration.FileSystemEnumerator`1..ctor(String directory, Boolean isNormalized, EnumerationOptions options)\\r\\n at System.IO.Directory.GetFiles(String path)\\r\\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_c3e4a5b6fs.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "c3e4a5b6-3333-4abc-9def-000000000003", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c3e4a5b6-3333-4abc-9def-000000000003\",\r\n \"StartTime\": \"2026-06-24T06:45:02.000Z\",\r\n \"EndTime\": \"2026-06-24T06:45:02.470Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"c3e4a5b6-3333-4abc-9def-000000000003\",\r\n \"Info\": \"Could not find a part of the path 'D:\\\\Reports\\\\2026\\\\Inbound'.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"List Inbound Files\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IO.DirectoryNotFoundException: Could not find a part of the path 'D:\\\\Reports\\\\2026\\\\Inbound'.\\r\\n at System.IO.Enumeration.FileSystemEnumerator`1.CreateDirectoryHandle(String path, Boolean ignoreNotFound)\\r\\n at System.IO.Enumeration.FileSystemEnumerator`1..ctor(String directory, Boolean isNormalized, EnumerationOptions options)\\r\\n at System.IO.Directory.GetFiles(String path)\\r\\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_c3e4a5b6fs.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_c3e4a5b6fs.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-24T06:45:02.000Z\",\r\n \"ReleaseName\": \"InboundFileSweep\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10003,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-24T06:45:02.470Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"c3e4a5b633334abc9def000000000003\",\r\n \"Id\": 4015003\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get c3e4a5b6-3333-4abc-9def-000000000003", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs c3e4a5b6-3333-4abc-9def-000000000003", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/task.yaml index 3ed89a0384..a96fa36c52 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/directory-not-found-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/README.md new file mode 100644 index 0000000000..af0e5e05d4 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/README.md @@ -0,0 +1,33 @@ +# If — Condition Compiler / Expression Error (Design-Time) + +Design-time (Studio) troubleshooting scenario for `System.Activities.Statements.If`. + +## What this scenario exercises + +Studio's Error List flags an `If` with `Compiler error(s) encountered processing expression` / +`Option Strict On disallows implicit conversions`. The agent must recognize this as a **type-mismatch +comparison in the `If` Condition** — `responseCode` is `System.Int32`, compared against the `String` +literal `"200"` — and prescribe comparing like types (`responseCode = 200`, or an explicit +`.ToString()` / `CInt` conversion). This is a design-time validation error, not a robot job fault and +not corrupt XAML / a missing package. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted VB project source: `Main.xaml` with an `If` whose Condition is `responseCode = "200"` where `responseCode` is declared `System.Int32`; `project.json` sets `expressionLanguage: VisualBasic` | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the `Integer`-vs-`String` type mismatch in the `If` Condition and the fix (compare + like types / convert explicitly — do NOT rebuild the `.xaml`, reinstall the package, or hunt for a + faulted job). + +Playbook: `references/runtime-exceptions/playbooks/if-condition-compiler-error.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/m/r/README.md deleted file mode 100644 index 245b32667f..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/m/r/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# If — Condition Compiler / Expression Error (Design-Time) - -Design-time (Studio) troubleshooting scenario for `System.Activities.Statements.If`. - -## What this scenario exercises - -Studio's Error List flags an `If` with `Compiler error(s) encountered processing expression` / -`Option Strict On disallows implicit conversions`. The agent must recognize this as a **type-mismatch -comparison in the `If` Condition** — `responseCode` is `System.Int32`, compared against the `String` -literal `"200"` — and prescribe comparing like types (`responseCode = 200`, or an explicit -`.ToString()` / `CInt` conversion). This is a design-time validation error, not a robot job fault and -not corrupt XAML / a missing package. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted VB project source: `Main.xaml` with an `If` whose Condition is `responseCode = "200"` where `responseCode` is declared `System.Int32`; `project.json` sets `expressionLanguage: VisualBasic` | -| `data/m/r/manifest.json` | `docsai ask` passthrough + permissive empty `unmocked_default` — no job/trace exists for a design-time error | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the `Integer`-vs-`String` type mismatch in the `If` Condition and the fix (compare - like types / convert explicitly — do NOT rebuild the `.xaml`, reinstall the package, or hunt for a - faulted job). - -Playbook: `references/runtime-exceptions/playbooks/if-condition-compiler-error.md`. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/m/r/manifest.json deleted file mode 100644 index 4626745300..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/m/r/manifest.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. Only docsai is proxied to the real CLI; any other uip call returns an empty array. This is a design-time Studio validation error with no Orchestrator job/log/trace, so no job fixtures exist. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/uip-fixture.json new file mode 100644 index 0000000000..bcb5501faa --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/data/uip-fixture.json @@ -0,0 +1,8 @@ +{ + "version": 1, + "responses": [], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/task.yaml index de506d41e1..ff8f2bd966 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-condition-compiler-error/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/README.md new file mode 100644 index 0000000000..e16ad7f0ea --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/README.md @@ -0,0 +1,28 @@ +# If Condition Key Not Found — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.Collections.Generic.KeyNotFoundException` thrown while resolving an **`If` activity +Condition** expression in `Main.xaml`. The agent runs the `uipath-troubleshoot` +skill against a `uip` CLI mock and must reach the same root cause as +`RESOLUTION.md`. + +## What the original session uncovered + +The `If` activity **Check Feature Enabled** in `Main.xaml` evaluates the condition `config["FeatureEnabled"] == "true"`, but the `config` dictionary only holds the key `Environment`. The key `FeatureEnabled` is absent, so resolving the `If` condition throws `System.Collections.Generic.KeyNotFoundException: The given key 'FeatureEnabled' was not present in the dictionary.` before either branch runs. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/key-not-found-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/README.md deleted file mode 100644 index 7dfbecad53..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# If Condition Key Not Found — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.Collections.Generic.KeyNotFoundException` thrown while resolving an **`If` activity -Condition** expression in `Main.xaml`. The agent runs the `uipath-troubleshoot` -skill against a `uip` CLI mock and must reach the same root cause as -`RESOLUTION.md`. - -## What the original session uncovered - -The `If` activity **Check Feature Enabled** in `Main.xaml` evaluates the condition `config["FeatureEnabled"] == "true"`, but the `config` dictionary only holds the key `Environment`. The key `FeatureEnabled` is absent, so resolving the `If` condition throws `System.Collections.Generic.KeyNotFoundException: The given key 'FeatureEnabled' was not present in the dictionary.` before either branch runs. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/key-not-found-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/cfe1400b92.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/cfe1400b92.json deleted file mode 100644 index 5c10f738a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/cfe1400b92.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-29T14:22:47.260Z", - "Message": "Check Feature Enabled: The given key 'FeatureEnabled' was not present in the dictionary.\r\nin Main.xaml\r\n at If \"Check Feature Enabled\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Collections.Generic.KeyNotFoundException\r\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_b8d9e0f1ft.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/f39f825d73.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/f39f825d73.json deleted file mode 100644 index d29588c078..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/f39f825d73.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "b8d9e0f1-8888-4abc-9def-000000000008", - "StartTime": "2026-06-29T14:22:47.000Z", - "EndTime": "2026-06-29T14:22:47.520Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "b8d9e0f1-8888-4abc-9def-000000000008", - "Info": "The given key 'FeatureEnabled' was not present in the dictionary.\r\n\r\nin Main.xaml\r\n at If \"Check Feature Enabled\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Collections.Generic.KeyNotFoundException: The given key 'FeatureEnabled' was not present in the dictionary.\r\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_b8d9e0f1ft.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-29T14:22:47.000Z", - "ReleaseName": "FeatureToggleGate", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10008, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-29T14:22:47.520Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "b8d9e0f188884abc9def000000000008", - "Id": 4016002 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/manifest.json deleted file mode 100644 index da32e1b1a6..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get b8d9e0f1-8888-4abc-9def-000000000008 --output json", - "file": "f39f825d73.json" - }, - { - "match": "or jobs logs b8d9e0f1-8888-4abc-9def-000000000008 --level Error --output json", - "file": "cfe1400b92.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get b8d9e0f1-8888-4abc-9def-000000000008", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs b8d9e0f1-8888-4abc-9def-000000000008", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/uip-fixture.json new file mode 100644 index 0000000000..60a7c7607b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "b8d9e0f1-8888-4abc-9def-000000000008", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-29T14:22:47.260Z\",\r\n \"Message\": \"Check Feature Enabled: The given key 'FeatureEnabled' was not present in the dictionary.\\r\\nin Main.xaml\\r\\n at If \\\"Check Feature Enabled\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Collections.Generic.KeyNotFoundException\\r\\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\\r\\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_b8d9e0f1ft.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "b8d9e0f1-8888-4abc-9def-000000000008", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"b8d9e0f1-8888-4abc-9def-000000000008\",\r\n \"StartTime\": \"2026-06-29T14:22:47.000Z\",\r\n \"EndTime\": \"2026-06-29T14:22:47.520Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"b8d9e0f1-8888-4abc-9def-000000000008\",\r\n \"Info\": \"The given key 'FeatureEnabled' was not present in the dictionary.\\r\\n\\r\\nin Main.xaml\\r\\n at If \\\"Check Feature Enabled\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Collections.Generic.KeyNotFoundException: The given key 'FeatureEnabled' was not present in the dictionary.\\r\\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\\r\\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_b8d9e0f1ft.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_b8d9e0f1ft.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-29T14:22:47.000Z\",\r\n \"ReleaseName\": \"FeatureToggleGate\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10008,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-29T14:22:47.520Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"b8d9e0f188884abc9def000000000008\",\r\n \"Id\": 4016002\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get b8d9e0f1-8888-4abc-9def-000000000008", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs b8d9e0f1-8888-4abc-9def-000000000008", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/task.yaml index 8c6899274a..09ca12e419 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-key-not-found-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/README.md new file mode 100644 index 0000000000..9579637b81 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/README.md @@ -0,0 +1,28 @@ +# If Condition Null Reference — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.NullReferenceException` thrown while resolving an **`If` activity +Condition** expression in `Main.xaml`. The agent runs the `uipath-troubleshoot` +skill against a `uip` CLI mock and must reach the same root cause as +`RESOLUTION.md`. + +## What the original session uncovered + +The `If` activity **Check Status Is Yes** in `Main.xaml` evaluates the condition `status.ToString() == "yes"`, but `status` is `null` — the upstream `Assign` **Set Status** sets it to `null`. Resolving the `If` condition calls `.ToString()` on the null reference, throwing `System.NullReferenceException` before either branch runs. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/null-reference-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/65b1af4626.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/65b1af4626.json deleted file mode 100644 index b5f96153c5..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/65b1af4626.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a7c8e9f0-7777-4abc-9def-000000000007", - "StartTime": "2026-06-28T10:11:02.000Z", - "EndTime": "2026-06-28T10:11:02.480Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "a7c8e9f0-7777-4abc-9def-000000000007", - "Info": "Object reference not set to an instance of an object.\r\n\r\nin Main.xaml\r\n at If \"Check Status Is Yes\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.NullReferenceException: Object reference not set to an instance of an object.\r\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_a7c8e9f0er.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-28T10:11:02.000Z", - "ReleaseName": "EligibilityRouter", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10007, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-28T10:11:02.480Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a7c8e9f077774abc9def000000000007", - "Id": 4016001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/README.md deleted file mode 100644 index fa9fe28dba..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/README.md +++ /dev/null @@ -1,27 +0,0 @@ -# If Condition Null Reference — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.NullReferenceException` thrown while resolving an **`If` activity -Condition** expression in `Main.xaml`. The agent runs the `uipath-troubleshoot` -skill against a `uip` CLI mock and must reach the same root cause as -`RESOLUTION.md`. - -## What the original session uncovered - -The `If` activity **Check Status Is Yes** in `Main.xaml` evaluates the condition `status.ToString() == "yes"`, but `status` is `null` — the upstream `Assign` **Set Status** sets it to `null`. Resolving the `If` condition calls `.ToString()` on the null reference, throwing `System.NullReferenceException` before either branch runs. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/null-reference-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/c97ab3a0a8.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/c97ab3a0a8.json deleted file mode 100644 index 1c6dc749e3..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/c97ab3a0a8.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-28T10:11:02.230Z", - "Message": "Check Status Is Yes: Object reference not set to an instance of an object.\r\nin Main.xaml\r\n at If \"Check Status Is Yes\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.NullReferenceException\r\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_a7c8e9f0er.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/manifest.json deleted file mode 100644 index 360994b554..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get a7c8e9f0-7777-4abc-9def-000000000007 --output json", - "file": "65b1af4626.json" - }, - { - "match": "or jobs logs a7c8e9f0-7777-4abc-9def-000000000007 --level Error --output json", - "file": "c97ab3a0a8.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get a7c8e9f0-7777-4abc-9def-000000000007", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs a7c8e9f0-7777-4abc-9def-000000000007", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/uip-fixture.json new file mode 100644 index 0000000000..d07c465dbc --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "a7c8e9f0-7777-4abc-9def-000000000007", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-28T10:11:02.230Z\",\r\n \"Message\": \"Check Status Is Yes: Object reference not set to an instance of an object.\\r\\nin Main.xaml\\r\\n at If \\\"Check Status Is Yes\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_a7c8e9f0er.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a7c8e9f0-7777-4abc-9def-000000000007", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a7c8e9f0-7777-4abc-9def-000000000007\",\r\n \"StartTime\": \"2026-06-28T10:11:02.000Z\",\r\n \"EndTime\": \"2026-06-28T10:11:02.480Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"a7c8e9f0-7777-4abc-9def-000000000007\",\r\n \"Info\": \"Object reference not set to an instance of an object.\\r\\n\\r\\nin Main.xaml\\r\\n at If \\\"Check Status Is Yes\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.NullReferenceException: Object reference not set to an instance of an object.\\r\\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_a7c8e9f0er.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_a7c8e9f0er.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-28T10:11:02.000Z\",\r\n \"ReleaseName\": \"EligibilityRouter\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10007,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-28T10:11:02.480Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a7c8e9f077774abc9def000000000007\",\r\n \"Id\": 4016001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get a7c8e9f0-7777-4abc-9def-000000000007", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs a7c8e9f0-7777-4abc-9def-000000000007", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/task.yaml index be4552764f..387045e94f 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-null-reference-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/README.md new file mode 100644 index 0000000000..4a30ddd5cd --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/README.md @@ -0,0 +1,33 @@ +# If — Wrong Branch Taken at Runtime (Case-Sensitive Comparison) + +Runtime troubleshooting scenario for `System.Activities.Statements.If` — a silent logic fault. + +## What this scenario exercises + +An unattended job completes **Successful** but produces the wrong outcome: every approved order is +routed to the manual review queue. The agent must trace it to the `If` "Check approval decision", whose +Condition `decision = "Approved"` is a case-sensitive VB string comparison — the source system sent +`APPROVED`, so `"APPROVED" = "Approved"` is `False` and the Else (manual review) branch runs. The fix is +a case-insensitive, trimmed comparison. There is no exception — the agent must reason from the branch +that ran vs the branch expected, not from an error string. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | crafted VB project source: `Main.xaml` with an `If` "Check approval decision" whose Condition is `decision = "Approved"` (exact-case, no `.Trim()`), Else branch logs "Routing to manual review queue" | + +## Success criteria + +Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): + +- Agent invoked the `uipath-troubleshoot` skill. +- Agent identified the case-sensitive `If` comparison as why the approved order took the Else branch, + and the fix (case-insensitive + trimmed comparison) — and did NOT misdiagnose it as a job fault or a + bad input value. + +Playbook: `references/runtime-exceptions/playbooks/if-wrong-branch.md`. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/3db6b96527.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/3db6b96527.json deleted file mode 100644 index b1d8b8639f..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/3db6b96527.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "c9a4e2f1-7b53-4d81-a260-3f8e1c6d0b49", - "StartTime": "2026-07-19T06:20:03.410Z", - "EndTime": "2026-07-19T06:20:05.120Z", - "State": "Successful", - "JobPriority": "Normal", - "Source": "Schedule", - "SourceType": "Schedule", - "Info": "Job completed", - "CreationTime": "2026-07-19T06:20:03.410Z", - "ReleaseName": "OrderApprovalRouter", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 1500287, - "FolderKey": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "ProcessType": "Process", - "LocalSystemAccount": "UIPATH\\svc_robot", - "OrchestratorUserIdentity": "original_email@test.com", - "LastModificationTime": "2026-07-19T06:20:05.120Z", - "JobError": {}, - "ErrorCode": null, - "TraceId": "c9a4e2f17b534d81a2603f8e1c6d0b49", - "Id": 66923456 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/701433db1c.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/701433db1c.json deleted file mode 100644 index 5eb35fc27c..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/701433db1c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "Result": "Success", - "Code": "JobHistory", - "Data": [ - { - "State": "Successful", - "Timestamp": "2026-07-19T06:20:05.120Z", - "JobId": 66923456 - }, - { - "State": "Running", - "Timestamp": "2026-07-19T06:20:03.870Z", - "JobId": 66923456 - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/75949d659e.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/75949d659e.json deleted file mode 100644 index 3bda31af0e..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/75949d659e.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "Result": "Success", - "Code": "JobList", - "Data": [ - { - "Key": "c9a4e2f1-7b53-4d81-a260-3f8e1c6d0b49", - "State": "Successful", - "ProcessName": "OrderApprovalRouter", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-19T06:20:03.410Z", - "EndTime": "2026-07-19T06:20:05.120Z", - "CreationTime": "2026-07-19T06:20:03.410Z" - }, - { - "Key": "4b8d1f60-2c37-4a95-8e01-6d3a9c2f7b18", - "State": "Successful", - "ProcessName": "OrderApprovalRouter", - "Source": "Schedule", - "Type": "Unattended", - "Priority": "Normal", - "Machine": "MOCK-HOST", - "StartTime": "2026-07-18T06:20:04.220Z", - "EndTime": "2026-07-18T06:20:06.010Z", - "CreationTime": "2026-07-18T06:20:04.220Z" - } - ], - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/README.md deleted file mode 100644 index fbff2bbd02..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/README.md +++ /dev/null @@ -1,32 +0,0 @@ -# If — Wrong Branch Taken at Runtime (Case-Sensitive Comparison) - -Runtime troubleshooting scenario for `System.Activities.Statements.If` — a silent logic fault. - -## What this scenario exercises - -An unattended job completes **Successful** but produces the wrong outcome: every approved order is -routed to the manual review queue. The agent must trace it to the `If` "Check approval decision", whose -Condition `decision = "Approved"` is a case-sensitive VB string comparison — the source system sent -`APPROVED`, so `"APPROVED" = "Approved"` is `False` and the Else (manual review) branch runs. The fix is -a case-insensitive, trimmed comparison. There is no exception — the agent must reason from the branch -that ran vs the branch expected, not from an error string. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `m/uip` | shared manifest-driven mock dispatcher from `../../_shared/mock_template/` | -| `process/` | crafted VB project source: `Main.xaml` with an `If` "Check approval decision" whose Condition is `decision = "Approved"` (exact-case, no `.Trim()`), Else branch logs "Routing to manual review queue" | -| `data/m/r/*.json` | **Successful**-job fixtures — `job-get` state `Successful` (no error code); `job-logs` show `Decision from source system: APPROVED` immediately followed by `Routing to manual review queue` (approved value → wrong branch); `job-traces` empty | -| `data/m/r/manifest.json` | `docsai ask` passthrough + `or jobs/folders` fixtures + permissive empty `unmocked_default` | - -## Success criteria - -Scores the **conclusion**, not the trajectory (`skill_triggered` + `llm_judge` against `RESOLUTION.md`): - -- Agent invoked the `uipath-troubleshoot` skill. -- Agent identified the case-sensitive `If` comparison as why the approved order took the Else branch, - and the fix (case-insensitive + trimmed comparison) — and did NOT misdiagnose it as a job fault or a - bad input value. - -Playbook: `references/runtime-exceptions/playbooks/if-wrong-branch.md`. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/b49ae1b1b0.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/b49ae1b1b0.json deleted file mode 100644 index 027a96abfc..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/b49ae1b1b0.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "Result": "Success", - "Code": "JobTraces", - "Data": [] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/b8d18266dc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/b8d18266dc.json deleted file mode 100644 index 8ca5287150..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/b8d18266dc.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Information", - "TimeStamp": "2026-07-19T06:20:04.980Z", - "Message": "Routing to manual review queue" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-19T06:20:04.560Z", - "Message": "Decision from source system: APPROVED" - }, - { - "Level": "Information", - "TimeStamp": "2026-07-19T06:20:03.910Z", - "Message": "OrderApprovalRouter execution started" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/ee2628e6b6.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/ee2628e6b6.json deleted file mode 100644 index f71a080fc8..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/ee2628e6b6.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/manifest.json deleted file mode 100644 index 2abd6ea336..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/m/r/manifest.json +++ /dev/null @@ -1,39 +0,0 @@ -{ - "version": 2, - "_doc": "Mock dispatch table. docsai proxied to the real CLI; job/folder lookups return canned Successful-job fixtures (the fault is a silent wrong-branch, not an exception); any other uip call returns an empty array. Matching is plain substring (first match wins); patterns are quote-agnostic.", - "rules": [ - { - "match": "docsai ask", - "passthrough": true - }, - { - "match": "or folders list", - "file": "ee2628e6b6.json" - }, - { - "match": "or jobs list", - "file": "75949d659e.json" - }, - { - "match": "or jobs logs", - "file": "b8d18266dc.json" - }, - { - "match": "or jobs history", - "file": "701433db1c.json" - }, - { - "match": "or jobs traces", - "file": "b49ae1b1b0.json" - }, - { - "match": "or jobs get", - "file": "3db6b96527.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched returns an empty array so the agent can explore without aborting." - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/uip-fixture.json new file mode 100644 index 0000000000..4ebc94f6ed --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/data/uip-fixture.json @@ -0,0 +1,69 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"6a0d3f92-4c81-4e57-9b28-1f5a0c8e3d67\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobList\",\r\n \"Data\": [\r\n {\r\n \"Key\": \"c9a4e2f1-7b53-4d81-a260-3f8e1c6d0b49\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"OrderApprovalRouter\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-19T06:20:03.410Z\",\r\n \"EndTime\": \"2026-07-19T06:20:05.120Z\",\r\n \"CreationTime\": \"2026-07-19T06:20:03.410Z\"\r\n },\r\n {\r\n \"Key\": \"4b8d1f60-2c37-4a95-8e01-6d3a9c2f7b18\",\r\n \"State\": \"Successful\",\r\n \"ProcessName\": \"OrderApprovalRouter\",\r\n \"Source\": \"Schedule\",\r\n \"Type\": \"Unattended\",\r\n \"Priority\": \"Normal\",\r\n \"Machine\": \"MOCK-HOST\",\r\n \"StartTime\": \"2026-07-18T06:20:04.220Z\",\r\n \"EndTime\": \"2026-07-18T06:20:06.010Z\",\r\n \"CreationTime\": \"2026-07-18T06:20:04.220Z\"\r\n }\r\n ],\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T06:20:04.980Z\",\r\n \"Message\": \"Routing to manual review queue\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T06:20:04.560Z\",\r\n \"Message\": \"Decision from source system: APPROVED\"\r\n },\r\n {\r\n \"Level\": \"Information\",\r\n \"TimeStamp\": \"2026-07-19T06:20:03.910Z\",\r\n \"Message\": \"OrderApprovalRouter execution started\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "history" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobHistory\",\r\n \"Data\": [\r\n {\r\n \"State\": \"Successful\",\r\n \"Timestamp\": \"2026-07-19T06:20:05.120Z\",\r\n \"JobId\": 66923456\r\n },\r\n {\r\n \"State\": \"Running\",\r\n \"Timestamp\": \"2026-07-19T06:20:03.870Z\",\r\n \"JobId\": 66923456\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "traces" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobTraces\",\r\n \"Data\": []\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"c9a4e2f1-7b53-4d81-a260-3f8e1c6d0b49\",\r\n \"StartTime\": \"2026-07-19T06:20:03.410Z\",\r\n \"EndTime\": \"2026-07-19T06:20:05.120Z\",\r\n \"State\": \"Successful\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Schedule\",\r\n \"SourceType\": \"Schedule\",\r\n \"Info\": \"Job completed\",\r\n \"CreationTime\": \"2026-07-19T06:20:03.410Z\",\r\n \"ReleaseName\": \"OrderApprovalRouter\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 1500287,\r\n \"FolderKey\": \"a2f18c53-9d07-4b26-8e41-6c0b3a9f7d28\",\r\n \"ProcessType\": \"Process\",\r\n \"LocalSystemAccount\": \"UIPATH\\\\svc_robot\",\r\n \"OrchestratorUserIdentity\": \"original_email@test.com\",\r\n \"LastModificationTime\": \"2026-07-19T06:20:05.120Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": null,\r\n \"TraceId\": \"c9a4e2f17b534d81a2603f8e1c6d0b49\",\r\n \"Id\": 66923456\r\n }\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + } +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/task.yaml index af6109ff98..933ae46fb2 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/if-wrong-branch-case/task.yaml @@ -18,21 +18,14 @@ run_limits: sandbox: python: {} template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: ["m"] - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 + passthrough_argv_prefixes: + - [docsai, ask] reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/README.md new file mode 100644 index 0000000000..fd13e542a2 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/README.md @@ -0,0 +1,27 @@ +# Index Out Of Range Exception — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.IndexOutOfRangeException` thrown from an `Assign` expression in +`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` +CLI mock and must reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Assign` **Extract Region Segment** in `Main.xaml` reads `parts[3]`, but `parts` came from `"INV-2026".Split('-')` which yields only 2 elements (indexes 0 and 1). Index 3 is past the end, so the array access throws `System.IndexOutOfRangeException: Index was outside the bounds of the array.` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/index-out-of-range-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/4a718393d4.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/4a718393d4.json deleted file mode 100644 index cf0ac91ca9..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/4a718393d4.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-25T11:20:55.180Z", - "Message": "Extract Region Segment: Index was outside the bounds of the array.\r\nin Main.xaml\r\n at Assign \"Extract Region Segment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.IndexOutOfRangeException\r\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_d4f5b6c7cp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/README.md deleted file mode 100644 index d09610b516..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Index Out Of Range Exception — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.IndexOutOfRangeException` thrown from an `Assign` expression in -`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` -CLI mock and must reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Assign` **Extract Region Segment** in `Main.xaml` reads `parts[3]`, but `parts` came from `"INV-2026".Split('-')` which yields only 2 elements (indexes 0 and 1). Index 3 is past the end, so the array access throws `System.IndexOutOfRangeException: Index was outside the bounds of the array.` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/index-out-of-range-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/b11c821a51.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/b11c821a51.json deleted file mode 100644 index ef0eb22d9e..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/b11c821a51.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "d4f5b6c7-4444-4abc-9def-000000000004", - "StartTime": "2026-06-25T11:20:55.000Z", - "EndTime": "2026-06-25T11:20:55.400Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "d4f5b6c7-4444-4abc-9def-000000000004", - "Info": "Index was outside the bounds of the array.\r\n\r\nin Main.xaml\r\n at Assign \"Extract Region Segment\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.IndexOutOfRangeException: Index was outside the bounds of the array.\r\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_d4f5b6c7cp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-25T11:20:55.000Z", - "ReleaseName": "InvoiceCodeParser", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10004, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-25T11:20:55.400Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "d4f5b6c744444abc9def000000000004", - "Id": 4015004 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/manifest.json deleted file mode 100644 index 071f33f0e9..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get d4f5b6c7-4444-4abc-9def-000000000004 --output json", - "file": "b11c821a51.json" - }, - { - "match": "or jobs logs d4f5b6c7-4444-4abc-9def-000000000004 --level Error --output json", - "file": "4a718393d4.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get d4f5b6c7-4444-4abc-9def-000000000004", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs d4f5b6c7-4444-4abc-9def-000000000004", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/uip-fixture.json new file mode 100644 index 0000000000..5865056f03 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "d4f5b6c7-4444-4abc-9def-000000000004", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T11:20:55.180Z\",\r\n \"Message\": \"Extract Region Segment: Index was outside the bounds of the array.\\r\\nin Main.xaml\\r\\n at Assign \\\"Extract Region Segment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.IndexOutOfRangeException\\r\\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_d4f5b6c7cp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "d4f5b6c7-4444-4abc-9def-000000000004", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"d4f5b6c7-4444-4abc-9def-000000000004\",\r\n \"StartTime\": \"2026-06-25T11:20:55.000Z\",\r\n \"EndTime\": \"2026-06-25T11:20:55.400Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"d4f5b6c7-4444-4abc-9def-000000000004\",\r\n \"Info\": \"Index was outside the bounds of the array.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Extract Region Segment\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.IndexOutOfRangeException: Index was outside the bounds of the array.\\r\\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_d4f5b6c7cp.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_d4f5b6c7cp.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-25T11:20:55.000Z\",\r\n \"ReleaseName\": \"InvoiceCodeParser\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10004,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-25T11:20:55.400Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"d4f5b6c744444abc9def000000000004\",\r\n \"Id\": 4015004\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get d4f5b6c7-4444-4abc-9def-000000000004", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs d4f5b6c7-4444-4abc-9def-000000000004", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/task.yaml index 35ddcbb61f..6ace584eef 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/index-out-of-range-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/README.md new file mode 100644 index 0000000000..56f112c073 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/README.md @@ -0,0 +1,27 @@ +# Invalid Operation Exception — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.InvalidOperationException` thrown from an `Assign` expression in +`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` +CLI mock and must reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Assign` **Find First Over 100** in `Main.xaml` runs the LINQ expression `numbers.First(n => n > 100)`, but no element of `numbers` (`{1, 2, 3}`) is greater than 100, so `First(predicate)` throws `System.InvalidOperationException: Sequence contains no matching element.` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/invalid-operation-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/README.md deleted file mode 100644 index 1149f427b8..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Invalid Operation Exception — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.InvalidOperationException` thrown from an `Assign` expression in -`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` -CLI mock and must reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Assign` **Find First Over 100** in `Main.xaml` runs the LINQ expression `numbers.First(n => n > 100)`, but no element of `numbers` (`{1, 2, 3}`) is greater than 100, so `First(predicate)` throws `System.InvalidOperationException: Sequence contains no matching element.` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/invalid-operation-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/ae1a53cbf5.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/ae1a53cbf5.json deleted file mode 100644 index e830055d9e..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/ae1a53cbf5.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1c2e3f4-1111-4abc-9def-000000000001", - "StartTime": "2026-06-22T09:14:11.120Z", - "EndTime": "2026-06-22T09:14:11.880Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "a1c2e3f4-1111-4abc-9def-000000000001", - "Info": "Sequence contains no matching element.\r\n\r\nin Main.xaml\r\n at Assign \"Find First Over 100\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.InvalidOperationException: Sequence contains no matching element.\r\n at System.Linq.ThrowHelper.ThrowNoMatchException()\r\n at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)\r\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_a1c2e3f4ip.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-22T09:14:11.120Z", - "ReleaseName": "OrderRecordLookup", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10001, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-22T09:14:11.880Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "a1c2e3f411114abc9def000000000001", - "Id": 4015001 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/d48d8eac3a.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/d48d8eac3a.json deleted file mode 100644 index 5d522b2b69..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/d48d8eac3a.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-22T09:14:11.560Z", - "Message": "Find First Over 100: Sequence contains no matching element.\r\nin Main.xaml\r\n at Assign \"Find First Over 100\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.InvalidOperationException\r\n at System.Linq.ThrowHelper.ThrowNoMatchException()\r\n at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)\r\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_a1c2e3f4ip.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/manifest.json deleted file mode 100644 index 5c43e49387..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get a1c2e3f4-1111-4abc-9def-000000000001 --output json", - "file": "ae1a53cbf5.json" - }, - { - "match": "or jobs logs a1c2e3f4-1111-4abc-9def-000000000001 --level Error --output json", - "file": "d48d8eac3a.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get a1c2e3f4-1111-4abc-9def-000000000001", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs a1c2e3f4-1111-4abc-9def-000000000001", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/uip-fixture.json new file mode 100644 index 0000000000..9a4e973f43 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "a1c2e3f4-1111-4abc-9def-000000000001", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-22T09:14:11.560Z\",\r\n \"Message\": \"Find First Over 100: Sequence contains no matching element.\\r\\nin Main.xaml\\r\\n at Assign \\\"Find First Over 100\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.InvalidOperationException\\r\\n at System.Linq.ThrowHelper.ThrowNoMatchException()\\r\\n at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)\\r\\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_a1c2e3f4ip.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "a1c2e3f4-1111-4abc-9def-000000000001", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1c2e3f4-1111-4abc-9def-000000000001\",\r\n \"StartTime\": \"2026-06-22T09:14:11.120Z\",\r\n \"EndTime\": \"2026-06-22T09:14:11.880Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"a1c2e3f4-1111-4abc-9def-000000000001\",\r\n \"Info\": \"Sequence contains no matching element.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Find First Over 100\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.InvalidOperationException: Sequence contains no matching element.\\r\\n at System.Linq.ThrowHelper.ThrowNoMatchException()\\r\\n at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)\\r\\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_a1c2e3f4ip.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_a1c2e3f4ip.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-22T09:14:11.120Z\",\r\n \"ReleaseName\": \"OrderRecordLookup\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10001,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-22T09:14:11.880Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a1c2e3f411114abc9def000000000001\",\r\n \"Id\": 4015001\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get a1c2e3f4-1111-4abc-9def-000000000001", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs a1c2e3f4-1111-4abc-9def-000000000001", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/task.yaml index 54e780f785..e6013725df 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/invalid-operation-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/README.md new file mode 100644 index 0000000000..acef97bf5c --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/README.md @@ -0,0 +1,27 @@ +# Key Not Found Exception — Faithful Replay + +This scenario replays a UiPath diagnostic investigation where an Orchestrator +job faulted with `System.Collections.Generic.KeyNotFoundException` thrown from an `Assign` expression in +`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` +CLI mock and must reach the same root cause as `RESOLUTION.md`. + +## What the original session uncovered + +The `Assign` **Read Api Base Url** in `Main.xaml` indexes `config["ApiBaseUrl"]`, but the `config` dictionary only holds the keys `Environment` and `Timeout`. The key `ApiBaseUrl` is absent, so the dictionary indexer throws `System.Collections.Generic.KeyNotFoundException: The given key 'ApiBaseUrl' was not present in the dictionary.` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/key-not-found-exception.md`) AND reached the same root cause as `RESOLUTION.md` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/46ac4679cc.json deleted file mode 100644 index e1bb9347a2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "test.user@example.com's workspace", - "Path": "test.user@example.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/71dd8edcd0.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/71dd8edcd0.json deleted file mode 100644 index 35325195ae..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/71dd8edcd0.json +++ /dev/null @@ -1,42 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "e5a6c7d8-5555-4abc-9def-000000000005", - "StartTime": "2026-06-26T08:30:18.000Z", - "EndTime": "2026-06-26T08:30:18.500Z", - "State": "Faulted", - "JobPriority": "Normal", - "SpecificPriorityValue": 45, - "Source": "Manual", - "SourceType": "Manual", - "BatchExecutionKey": "e5a6c7d8-5555-4abc-9def-000000000005", - "Info": "The given key 'ApiBaseUrl' was not present in the dictionary.\r\n\r\nin Main.xaml\r\n at Assign \"Read Api Base Url\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\nSystem.Collections.Generic.KeyNotFoundException: The given key 'ApiBaseUrl' was not present in the dictionary.\r\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_e5a6c7d8cr.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)", - "CreationTime": "2026-06-26T08:30:18.000Z", - "ReleaseName": "ConfigReader", - "Type": "Unattended", - "InputArguments": "{}", - "EnvironmentVariables": "", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "HasMediaRecorded": false, - "HasVideoRecorded": false, - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10005, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Reference": "", - "ProcessType": "Process", - "ResumeOnSameContext": false, - "LocalSystemAccount": "UIPATH\\svcRobot", - "OrchestratorUserIdentity": "test.user@example.com", - "RemoteControlAccess": "None", - "LastModificationTime": "2026-06-26T08:30:18.500Z", - "JobError": {}, - "ErrorCode": "Robot", - "TraceId": "e5a6c7d855554abc9def000000000005", - "Id": 4015005 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/README.md deleted file mode 100644 index ccc68455e4..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Key Not Found Exception — Faithful Replay - -This scenario replays a UiPath diagnostic investigation where an Orchestrator -job faulted with `System.Collections.Generic.KeyNotFoundException` thrown from an `Assign` expression in -`Main.xaml`. The agent runs the `uipath-troubleshoot` skill against a `uip` -CLI mock and must reach the same root cause as `RESOLUTION.md`. - -## What the original session uncovered - -The `Assign` **Read Api Base Url** in `Main.xaml` indexes `config["ApiBaseUrl"]`, but the `config` dictionary only holds the keys `Environment` and `Timeout`. The key `ApiBaseUrl` is absent, so the dictionary indexer throws `System.Collections.Generic.KeyNotFoundException: The given key 'ApiBaseUrl' was not present in the dictionary.` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project (`Main.xaml`) | -| `fixtures/mocks/responses/*.json` | recorded `uip or` stdout: folder list, faulted job, error logs | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook (`references/runtime-exceptions/playbooks/key-not-found-exception.md`) AND reached the same root cause as `RESOLUTION.md` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/d86ae543f6.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/d86ae543f6.json deleted file mode 100644 index 5d3cffa5f2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/d86ae543f6.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-26T08:30:18.250Z", - "Message": "Read Api Base Url: The given key 'ApiBaseUrl' was not present in the dictionary.\r\nin Main.xaml\r\n at Assign \"Read Api Base Url\"\r\n at Sequence \"Main Sequence\"\r\n at Main \"Main\"\r\n\r\n--- .NET stack trace ---\r\nSystem.Collections.Generic.KeyNotFoundException\r\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_e5a6c7d8cr.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/manifest.json deleted file mode 100644 index 355f4573e5..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/m/r/manifest.json +++ /dev/null @@ -1,40 +0,0 @@ -{ - "version": 2, - "_doc": "Hand-authored faithful-replay fixtures. Deterministic first-match rules; unmatched commands return an empty array.", - "rules": [ - { - "match": "or folders list --output json", - "file": "46ac4679cc.json" - }, - { - "match": "or jobs get e5a6c7d8-5555-4abc-9def-000000000005 --output json", - "file": "71dd8edcd0.json" - }, - { - "match": "or jobs logs e5a6c7d8-5555-4abc-9def-000000000005 --level Error --output json", - "file": "d86ae543f6.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any unmatched uip command returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get e5a6c7d8-5555-4abc-9def-000000000005", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs e5a6c7d8-5555-4abc-9def-000000000005", - "min": 0, - "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/uip-fixture.json new file mode 100644 index 0000000000..f9bb302e3e --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "e5a6c7d8-5555-4abc-9def-000000000005", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-26T08:30:18.250Z\",\r\n \"Message\": \"Read Api Base Url: The given key 'ApiBaseUrl' was not present in the dictionary.\\r\\nin Main.xaml\\r\\n at Assign \\\"Read Api Base Url\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.Collections.Generic.KeyNotFoundException\\r\\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\\r\\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_e5a6c7d8cr.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "e5a6c7d8-5555-4abc-9def-000000000005", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"e5a6c7d8-5555-4abc-9def-000000000005\",\r\n \"StartTime\": \"2026-06-26T08:30:18.000Z\",\r\n \"EndTime\": \"2026-06-26T08:30:18.500Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"SpecificPriorityValue\": 45,\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"BatchExecutionKey\": \"e5a6c7d8-5555-4abc-9def-000000000005\",\r\n \"Info\": \"The given key 'ApiBaseUrl' was not present in the dictionary.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Read Api Base Url\\\"\\r\\n at Sequence \\\"Main Sequence\\\"\\r\\n at Main \\\"Main\\\"\\r\\n\\r\\nSystem.Collections.Generic.KeyNotFoundException: The given key 'ApiBaseUrl' was not present in the dictionary.\\r\\n at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\\r\\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_e5a6c7d8cr.Main_Expressions.Main_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_e5a6c7d8cr.Main_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\",\r\n \"CreationTime\": \"2026-06-26T08:30:18.000Z\",\r\n \"ReleaseName\": \"ConfigReader\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"EnvironmentVariables\": \"\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"HasMediaRecorded\": false,\r\n \"HasVideoRecorded\": false,\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10005,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Reference\": \"\",\r\n \"ProcessType\": \"Process\",\r\n \"ResumeOnSameContext\": false,\r\n \"LocalSystemAccount\": \"UIPATH\\\\svcRobot\",\r\n \"OrchestratorUserIdentity\": \"test.user@example.com\",\r\n \"RemoteControlAccess\": \"None\",\r\n \"LastModificationTime\": \"2026-06-26T08:30:18.500Z\",\r\n \"JobError\": {},\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"e5a6c7d855554abc9def000000000005\",\r\n \"Id\": 4015005\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"test.user@example.com's workspace\",\r\n \"Path\": \"test.user@example.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get e5a6c7d8-5555-4abc-9def-000000000005", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs e5a6c7d8-5555-4abc-9def-000000000005", + "min": 0, + "description": "OPTIONAL — agent may pull error-level logs for additional detail; min:0 = tracked-but-not-required." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/task.yaml index 6ecd7a653a..58f7d96bc6 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/key-not-found-exception/task.yaml @@ -17,22 +17,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/README.md new file mode 100644 index 0000000000..80b6ed493a --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/README.md @@ -0,0 +1,36 @@ +# Null Reference Exception — Faithful Replay + +This scenario replays a real UiPath diagnostic investigation where the +agent reached a verified resolution. The fixtures are the verbatim +`uip` CLI responses captured from that session. + +## What the original session uncovered + +Root Cause: A C# input-argument expression on the `Copy File` activity in `CopyFile.xaml` dereferences a variable that the workflow itself sets to `null` before the activity runs. + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | frozen snapshot of the failing UiPath project | + +## Success criteria + +The test scores the **conclusion**, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` + +## Re-running the extraction + +If the source transcript or project changes, regenerate the scenario: + +```bash +python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ + --investigation --project --transcript \ + --scenario-name null-reference-exception --apply +``` + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/897a8be769.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/897a8be769.json deleted file mode 100644 index 4abdc6cd89..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/897a8be769.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "original_email@test.com's workspace", - "Path": "original_email@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/README.md deleted file mode 100644 index a5cf41d5d6..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/README.md +++ /dev/null @@ -1,35 +0,0 @@ -# Null Reference Exception — Faithful Replay - -This scenario replays a real UiPath diagnostic investigation where the -agent reached a verified resolution. The fixtures are the verbatim -`uip` CLI responses captured from that session. - -## What the original session uncovered - -Root Cause: A C# input-argument expression on the `Copy File` activity in `CopyFile.xaml` dereferences a variable that the workflow itself sets to `null` before the activity runs. - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | frozen snapshot of the failing UiPath project | -| `fixtures/mocks/responses/*.json` | real stdout extracted verbatim from the session transcript | -| `fixtures/mocks/responses/manifest.json` | dispatch table mapping each command pattern to its recorded fixture | - -## Success criteria - -The test scores the **conclusion**, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause as `RESOLUTION.md` - -## Re-running the extraction - -If the source transcript or project changes, regenerate the scenario: - -```bash -python tests/tasks/uipath-troubleshoot/_shared/scripts/generate_scenario.py \ - --investigation --project --transcript \ - --scenario-name null-reference-exception --apply -``` diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/b89dac2859.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/b89dac2859.json deleted file mode 100644 index 30d3b5fe93..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/b89dac2859.json +++ /dev/null @@ -1,9 +0,0 @@ -{ - - "Result": "ValidationError", - - "Message": "error: unknown option '--folder-key'", - - "Instructions": "Check command arguments and options. Use --help for usage information." - -} \ No newline at end of file diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/f6285946d0.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/f6285946d0.json deleted file mode 100644 index da2e8ebdba..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/f6285946d0.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "Result": "Success", - "Code": "JobLogs", - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-05-13T07:32:23.021Z", - "Message": "Copy File: Object reference not set to an instance of an object.\r\nin CopyFile.xaml\r\n at CopyFile \"Copy File\"\r\n at Sequence \"ERN\"\r\n at ERN \"ERN\"\r\n\r\n--- .NET stack trace ---\r\nSystem.NullReferenceException\r\n at Namespace_fmq3zcn67w.ERN_Expressions.ERN_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\r\n at Namespace_fmq3zcn67w.ERN_Expressions.ERN_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\r\n at Namespace_fmq3zcn67w.ERN_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\r\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/manifest.json deleted file mode 100644 index b43f6152d9..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/m/r/manifest.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "version": 2, - "_doc": "Auto-generated by generate_scenario.py - review fixtures + add expected_calls.", - "rules": [ - { - "match": "or folders list --output json", - "file": "897a8be769.json" - }, - { - "match": "or jobs get 77dc53dd-5c45-4a77-b39f-a52e9e7ef163 --output json", - "file": "b89dac2859.json", - "exit_code": 1 - }, - { - "match": "or jobs logs 77dc53dd-5c45-4a77-b39f-a52e9e7ef163 --level Error --output json", - "file": "f6285946d0.json" - } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched by `rules` returns an empty array. Lets the agent explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { - "pattern": "or folders list", - "min": 1, - "description": "Agent enumerates folders to locate the folder that owns the job under investigation." - }, - { - "pattern": "or jobs get 77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "min": 1, - "description": "Agent fetches the job's execution details." - }, - { - "pattern": "or jobs logs 77dc53dd-5c45-4a77-b39f-a52e9e7ef163", - "min": 0, - "description": "Agent reads error-level job logs for additional execution detail." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/uip-fixture.json new file mode 100644 index 0000000000..cbd4195402 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/data/uip-fixture.json @@ -0,0 +1,66 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "logs", + "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", + "--level", + "Error", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobLogs\",\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-05-13T07:32:23.021Z\",\r\n \"Message\": \"Copy File: Object reference not set to an instance of an object.\\r\\nin CopyFile.xaml\\r\\n at CopyFile \\\"Copy File\\\"\\r\\n at Sequence \\\"ERN\\\"\\r\\n at ERN \\\"ERN\\\"\\r\\n\\r\\n--- .NET stack trace ---\\r\\nSystem.NullReferenceException\\r\\n at Namespace_fmq3zcn67w.ERN_Expressions.ERN_Expressions_TypedDataContext2_ForReadOnly.__Expr2Get()\\r\\n at Namespace_fmq3zcn67w.ERN_Expressions.ERN_Expressions_TypedDataContext2_ForReadOnly.ValueType___Expr2Get()\\r\\n at Namespace_fmq3zcn67w.ERN_Expressions.InvokeExpression(Int32 expressionId, IList`1 locations, ActivityContext activityContext)\\r\\n at Microsoft.CSharp.Activities.CSharpValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.InternalTryPopulateArgumentValueOrScheduleExpression(RuntimeArgument argument, Int32 nextArgumentIndex, ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Boolean isDynamicUpdate)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "get", + "77dc53dd-5c45-4a77-b39f-a52e9e7ef163", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 1, + "stdout": "{\r\n\r\n \"Result\": \"ValidationError\",\r\n\r\n \"Message\": \"error: unknown option '--folder-key'\",\r\n\r\n \"Instructions\": \"Check command arguments and options. Use --help for usage information.\"\r\n\r\n}" + }, + { + "argv": [ + "or", + "folders", + "list", + "--output", + "json" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"original_email@test.com's workspace\",\r\n \"Path\": \"original_email@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the folder that owns the job under investigation." + }, + { + "pattern": "or jobs get 77dc53dd-5c45-4a77-b39f-a52e9e7ef163", + "min": 1, + "description": "Agent fetches the job's execution details." + }, + { + "pattern": "or jobs logs 77dc53dd-5c45-4a77-b39f-a52e9e7ef163", + "min": 0, + "description": "Agent reads error-level job logs for additional execution detail." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/task.yaml index 9a5ca3c362..f30ca1a54a 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/null-reference-exception/task.yaml @@ -18,22 +18,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/README.md new file mode 100644 index 0000000000..b20436fbc7 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/README.md @@ -0,0 +1,42 @@ +# VisualBasicValue Requires Compilation — Expression-Language Mismatch + +This scenario reproduces an Orchestrator job that faults with +`Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.` + +## What this scenario uncovers + +**Root Cause:** The `OrderRouter` project is `expressionLanguage: CSharp` / +`targetFramework: Windows`, but the `Assign 'Build Route Key'` value is a +`VisualBasicValue` node (a VB `&` concatenation) — a VB expression left in a C# +project, typically from pasting an activity copied out of a VB project. Modern +.NET projects disable runtime expression JIT, so the stray VB expression has no +compiled delegate and throws at run. The designer shows the workflow as valid. + +This maps to: +`skills/uipath-troubleshoot/references/runtime-exceptions/playbooks/visualbasicvalue-requires-compilation.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `OrderRouter` — C# / Windows project; `Assign 'Build Route Key'` value is a `` node | + +The decisive evidence is the mismatch: `project.json` declares +`expressionLanguage: CSharp`, yet `Main.xaml` carries a `VisualBasicValue` node. +The agent must read both the project file and the workflow source to identify it. + +## Success criteria + +The test scores the conclusion, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause + fix as `RESOLUTION.md` + +> **Note on fixtures.** Synthetic. Job key, folder key, and host are placeholders. +> The error is a robot/Orchestrator-runtime fault and is not reproducible via the +> local `uip rpa` CLI (which AOT-compiles expressions); the mocks supply the +> faulted-job evidence. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/267d18282a.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/267d18282a.json deleted file mode 100644 index c0085ed679..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/267d18282a.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "LogList", - "Pagination": { - "Returned": 1, - "Limit": 100, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-25T13:02:42.098Z", - "Message": "OrderRouter execution error: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run. (faulting activity: Assign 'Build Route Key' in Main.xaml)", - "RobotName": "MOCK-HOST", - "HostMachineName": "MOCK-HOST", - "ProcessName": "OrderRouter", - "JobKey": "bb22cc33-dd44-ee55-ff66-778899001122", - "RawMessage": "System.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/46ac4679cc.json deleted file mode 100644 index 27e34874bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "automation@test.com's workspace", - "Path": "automation@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/8579c46e94.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/8579c46e94.json deleted file mode 100644 index 93de27eef7..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/8579c46e94.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "bb22cc33-dd44-ee55-ff66-778899001122", - "StartTime": "2026-06-25T13:02:41.305Z", - "EndTime": "2026-06-25T13:02:42.110Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\r\n\r\nin Main.xaml\r\n at Assign \"Build Route Key\"\r\n at Sequence \"Route Order\"\r\n\r\nSystem.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\r\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\r\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)", - "CreationTime": "2026-06-25T13:02:41.305Z", - "ReleaseName": "OrderRouter", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10502, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "ErrorCode": "Robot", - "TraceId": "bb22cc33dd44ee55ff66778899001122", - "Id": 4019502 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/README.md deleted file mode 100644 index 71dbda9e46..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# VisualBasicValue Requires Compilation — Expression-Language Mismatch - -This scenario reproduces an Orchestrator job that faults with -`Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.` - -## What this scenario uncovers - -**Root Cause:** The `OrderRouter` project is `expressionLanguage: CSharp` / -`targetFramework: Windows`, but the `Assign 'Build Route Key'` value is a -`VisualBasicValue` node (a VB `&` concatenation) — a VB expression left in a C# -project, typically from pasting an activity copied out of a VB project. Modern -.NET projects disable runtime expression JIT, so the stray VB expression has no -compiled delegate and throws at run. The designer shows the workflow as valid. - -This maps to: -`skills/uipath-troubleshoot/references/runtime-exceptions/playbooks/visualbasicvalue-requires-compilation.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | `OrderRouter` — C# / Windows project; `Assign 'Build Route Key'` value is a `` node | -| `fixtures/mocks/responses/*.json` | synthetic canned `uip` responses; the faulted job's `Info` carries the `VisualBasicValue`1 requires compilation` error and the stack frame at `Assign 'Build Route Key'` | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The decisive evidence is the mismatch: `project.json` declares -`expressionLanguage: CSharp`, yet `Main.xaml` carries a `VisualBasicValue` node. -The agent must read both the project file and the workflow source to identify it. - -## Success criteria - -The test scores the conclusion, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause + fix as `RESOLUTION.md` - -> **Note on fixtures.** Synthetic. Job key, folder key, and host are placeholders. -> The error is a robot/Orchestrator-runtime fault and is not reproducible via the -> local `uip rpa` CLI (which AOT-compiles expressions); the mocks supply the -> faulted-job evidence. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/manifest.json deleted file mode 100644 index f02aac98e2..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/m/r/manifest.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the expression-language-mismatch branch of the 'VisualBasicValue`1 requires compilation' runtime-exceptions playbook. The OrderRouter project is expressionLanguage CSharp / targetFramework Windows, but the Assign 'Build Route Key' value is a VisualBasicValue node (VB '&' concatenation) left over from a paste out of a VB project. Modern .NET runtime JIT is disabled, so the stray VB expression cannot compile at run. Decisive evidence: project.json expressionLanguage=CSharp combined with the node in process/Main.xaml. Expected chain: folders list -> jobs get -> jobs logs --level Error -> read process/project.json + Main.xaml.", - "rules": [ - { "match": "or folders list", "file": "46ac4679cc.json" }, - { "match": "or jobs get bb22cc33-dd44-ee55-ff66-778899001122", "file": "8579c46e94.json" }, - { "match": "or jobs logs bb22cc33-dd44-ee55-ff66-778899001122", "file": "267d18282a.json" } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched above returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { "pattern": "or folders list", "min": 1, "description": "Agent enumerates folders to locate the Shared folder where the faulted job lives." }, - { "pattern": "or jobs get bb22cc33-dd44-ee55-ff66-778899001122", "min": 1, "description": "Agent fetches the faulted job to read the 'requires compilation' error and the stack frame pointing at the Assign 'Build Route Key'." }, - { "pattern": "or jobs logs bb22cc33-dd44-ee55-ff66-778899001122", "min": 0, "description": "OPTIONAL - error-level logs corroborate the error; the job-get Info already carries it." } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/uip-fixture.json new file mode 100644 index 0000000000..5ef0c5c66b --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/data/uip-fixture.json @@ -0,0 +1,58 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "get", + "bb22cc33-dd44-ee55-ff66-778899001122" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"bb22cc33-dd44-ee55-ff66-778899001122\",\r\n \"StartTime\": \"2026-06-25T13:02:41.305Z\",\r\n \"EndTime\": \"2026-06-25T13:02:42.110Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Build Route Key\\\"\\r\\n at Sequence \\\"Route Order\\\"\\r\\n\\r\\nSystem.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\\r\\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\\r\\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\",\r\n \"CreationTime\": \"2026-06-25T13:02:41.305Z\",\r\n \"ReleaseName\": \"OrderRouter\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10502,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"bb22cc33dd44ee55ff66778899001122\",\r\n \"Id\": 4019502\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "bb22cc33-dd44-ee55-ff66-778899001122" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LogList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-25T13:02:42.098Z\",\r\n \"Message\": \"OrderRouter execution error: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run. (faulting activity: Assign 'Build Route Key' in Main.xaml)\",\r\n \"RobotName\": \"MOCK-HOST\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"OrderRouter\",\r\n \"JobKey\": \"bb22cc33-dd44-ee55-ff66-778899001122\",\r\n \"RawMessage\": \"System.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"automation@test.com's workspace\",\r\n \"Path\": \"automation@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the Shared folder where the faulted job lives." + }, + { + "pattern": "or jobs get bb22cc33-dd44-ee55-ff66-778899001122", + "min": 1, + "description": "Agent fetches the faulted job to read the 'requires compilation' error and the stack frame pointing at the Assign 'Build Route Key'." + }, + { + "pattern": "or jobs logs bb22cc33-dd44-ee55-ff66-778899001122", + "min": 0, + "description": "OPTIONAL - error-level logs corroborate the error; the job-get Info already carries it." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/task.yaml index 58356939a1..09be0ff0d2 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-csharp-mismatch/task.yaml @@ -25,22 +25,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/README.md new file mode 100644 index 0000000000..2d7503e6e5 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/README.md @@ -0,0 +1,42 @@ +# VisualBasicValue Requires Compilation — Smart Quotes + +This scenario reproduces an Orchestrator job that faults with +`Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.` + +## What this scenario uncovers + +**Root Cause:** The `Assign 'Build Greeting'` value expression in `Main.xaml` was +pasted from rich text and uses curly/smart double quotes (U+201C and U+201D) +instead of straight ASCII quotes (U+0022). The expression could not be compiled +ahead-of-time; the project is `VisualBasic` / `targetFramework: Windows` (.NET 6+, +runtime JIT disabled), so the uncompiled expression throws at run. The designer +shows the workflow as valid. + +This maps to: +`skills/uipath-troubleshoot/references/runtime-exceptions/playbooks/visualbasicvalue-requires-compilation.md` + +## How this test reproduces it + +| Layer | Source | +|---|---| +| `process/` | `InvoiceNotifier` — VB / Windows project; `Assign 'Build Greeting'` value uses smart quotes | + +The decisive evidence is the smart double quotes (U+201C/U+201D) in the +`Assign 'Build Greeting'` value in `process/Main.xaml` — the agent must read the +workflow source, not just the job logs, to identify them. + +## Success criteria + +The test scores the conclusion, not the trajectory: + +- Agent invoked the `uipath-troubleshoot` skill +- Agent matched the correct playbook AND reached the same root cause + fix as `RESOLUTION.md` + +> **Note on fixtures.** Synthetic. Job key, folder key, and host are placeholders. +> The error is a robot/Orchestrator-runtime fault and is not reproducible via the +> local `uip rpa` CLI (which AOT-compiles expressions); the mocks supply the +> faulted-job evidence. + +## Fixture isolation + +`data/uip-fixture.json` is a finite command/response map mounted only into coder-eval's host-side protected mock service. The evaluated agent receives a bounded `uip` client and cannot read the fixture or mock implementation. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/368f2e84cf.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/368f2e84cf.json deleted file mode 100644 index 01aa090e6c..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/368f2e84cf.json +++ /dev/null @@ -1,30 +0,0 @@ -{ - "Result": "Success", - "Code": "JobGet", - "Data": { - "Key": "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", - "StartTime": "2026-06-24T09:14:08.117Z", - "EndTime": "2026-06-24T09:14:08.992Z", - "State": "Faulted", - "JobPriority": "Normal", - "Source": "Manual", - "SourceType": "Manual", - "Info": "Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\r\n\r\nin Main.xaml\r\n at Assign \"Build Greeting\"\r\n at Sequence \"Notify Customer\"\r\n\r\nSystem.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\r\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\r\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\r\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\r\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\r\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\r\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)", - "CreationTime": "2026-06-24T09:14:08.117Z", - "ReleaseName": "InvoiceNotifier", - "Type": "Unattended", - "InputArguments": "{}", - "OutputArguments": "{}", - "HostMachineName": "MOCK-HOST", - "RuntimeType": "Unattended", - "RequiresUserInteraction": false, - "ReleaseVersionId": 10204, - "EntryPointPath": "Main.xaml", - "OrganizationUnitId": 28927, - "FolderKey": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "ProcessType": "Process", - "ErrorCode": "Robot", - "TraceId": "a1b2c3d4e5f64a7b8c9d0e1f2a3b4c5d", - "Id": 4019338 - } -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/4667db66cb.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/4667db66cb.json deleted file mode 100644 index caa48f8082..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/4667db66cb.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "Result": "Success", - "Code": "LogList", - "Pagination": { - "Returned": 1, - "Limit": 100, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Level": "Error", - "TimeStamp": "2026-06-24T09:14:08.981Z", - "Message": "InvoiceNotifier execution error: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run. (faulting activity: Assign 'Build Greeting' in Main.xaml)", - "RobotName": "MOCK-HOST", - "HostMachineName": "MOCK-HOST", - "ProcessName": "InvoiceNotifier", - "JobKey": "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", - "RawMessage": "System.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run." - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/46ac4679cc.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/46ac4679cc.json deleted file mode 100644 index 27e34874bd..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/46ac4679cc.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "Result": "Success", - "Code": "FolderList", - "Pagination": { - "Returned": 2, - "Limit": 50, - "Offset": 0, - "HasMore": false - }, - "Data": [ - { - "Key": "dea5d8d6-ca39-4dbf-b4a5-3516bad38866", - "Name": "automation@test.com's workspace", - "Path": "automation@test.com's workspace", - "Description": "", - "Type": "Personal", - "ParentKey": "" - }, - { - "Key": "1965a46b-db4e-469e-aaaa-7e0b379cb34d", - "Name": "Shared", - "Path": "Shared", - "Description": "", - "Type": "Standard", - "ParentKey": "" - } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/README.md b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/README.md deleted file mode 100644 index 31e5695a52..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/README.md +++ /dev/null @@ -1,41 +0,0 @@ -# VisualBasicValue Requires Compilation — Smart Quotes - -This scenario reproduces an Orchestrator job that faults with -`Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.` - -## What this scenario uncovers - -**Root Cause:** The `Assign 'Build Greeting'` value expression in `Main.xaml` was -pasted from rich text and uses curly/smart double quotes (U+201C and U+201D) -instead of straight ASCII quotes (U+0022). The expression could not be compiled -ahead-of-time; the project is `VisualBasic` / `targetFramework: Windows` (.NET 6+, -runtime JIT disabled), so the uncompiled expression throws at run. The designer -shows the workflow as valid. - -This maps to: -`skills/uipath-troubleshoot/references/runtime-exceptions/playbooks/visualbasicvalue-requires-compilation.md` - -## How this test reproduces it - -| Layer | Source | -|---|---| -| `mocks/uip` + `mocks/uip.cmd` | shared from `../_shared/mock_template/` (manifest-driven Python dispatcher) | -| `process/` | `InvoiceNotifier` — VB / Windows project; `Assign 'Build Greeting'` value uses smart quotes | -| `fixtures/mocks/responses/*.json` | synthetic canned `uip` responses; the faulted job's `Info` carries the `VisualBasicValue`1 requires compilation` error and the stack frame at `Assign 'Build Greeting'` | -| `fixtures/mocks/responses/manifest.json` | dispatch table | - -The decisive evidence is the smart double quotes (U+201C/U+201D) in the -`Assign 'Build Greeting'` value in `process/Main.xaml` — the agent must read the -workflow source, not just the job logs, to identify them. - -## Success criteria - -The test scores the conclusion, not the trajectory: - -- Agent invoked the `uipath-troubleshoot` skill -- Agent matched the correct playbook AND reached the same root cause + fix as `RESOLUTION.md` - -> **Note on fixtures.** Synthetic. Job key, folder key, and host are placeholders. -> The error is a robot/Orchestrator-runtime fault and is not reproducible via the -> local `uip rpa` CLI (which AOT-compiles expressions); the mocks supply the -> faulted-job evidence. diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/manifest.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/manifest.json deleted file mode 100644 index f29f536a6e..0000000000 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/m/r/manifest.json +++ /dev/null @@ -1,19 +0,0 @@ -{ - "version": 2, - "_doc": "Synthetic fixture set for the 'Expression Activity type VisualBasicValue`1 requires compilation in order to run' branch of the runtime-exceptions playbook. The InvoiceNotifier job faults at the Assign 'Build Greeting' because its value expression was pasted from rich text and contains curly/smart double quotes (U+201C/U+201D) instead of straight ASCII quotes, so the expression was not AOT-compiled; the modern Windows (.NET) runtime has JIT disabled and throws at run. Decisive evidence: the smart quotes in process/Main.xaml's Assign 'Build Greeting' value. Expected chain: folders list -> jobs get -> jobs logs --level Error -> read process/Main.xaml.", - "rules": [ - { "match": "or folders list", "file": "46ac4679cc.json" }, - { "match": "or jobs get a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", "file": "368f2e84cf.json" }, - { "match": "or jobs logs a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", "file": "4667db66cb.json" } - ], - "unmocked_default": { - "response": "[]\n", - "exit_code": 0, - "_doc": "Permissive fallback: any uip command not matched above returns an empty array so the agent can explore beyond the recorded path without aborting." - }, - "expected_calls": [ - { "pattern": "or folders list", "min": 1, "description": "Agent enumerates folders to locate the Shared folder where the faulted job lives." }, - { "pattern": "or jobs get a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", "min": 1, "description": "Agent fetches the faulted job to read the 'requires compilation' error and the stack frame pointing at the Assign 'Build Greeting'." }, - { "pattern": "or jobs logs a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", "min": 0, "description": "OPTIONAL - error-level logs corroborate the error; the job-get Info already carries it." } - ] -} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/uip-fixture.json b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/uip-fixture.json new file mode 100644 index 0000000000..95e700ca34 --- /dev/null +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/data/uip-fixture.json @@ -0,0 +1,58 @@ +{ + "version": 1, + "responses": [ + { + "argv": [ + "or", + "jobs", + "get", + "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"JobGet\",\r\n \"Data\": {\r\n \"Key\": \"a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"StartTime\": \"2026-06-24T09:14:08.117Z\",\r\n \"EndTime\": \"2026-06-24T09:14:08.992Z\",\r\n \"State\": \"Faulted\",\r\n \"JobPriority\": \"Normal\",\r\n \"Source\": \"Manual\",\r\n \"SourceType\": \"Manual\",\r\n \"Info\": \"Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\\r\\n\\r\\nin Main.xaml\\r\\n at Assign \\\"Build Greeting\\\"\\r\\n at Sequence \\\"Notify Customer\\\"\\r\\n\\r\\nSystem.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\\r\\n at System.Activities.Expressions.CompiledExpressionInvoker.InvokeExpression(ActivityContext activityContext)\\r\\n at Microsoft.VisualBasic.Activities.VisualBasicValue`1.Execute(CodeActivityContext context)\\r\\n at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)\\r\\n at System.Activities.Runtime.ActivityExecutor.ExecuteInResolutionContext[T](ActivityInstance parentInstance, Activity`1 expressionActivity)\\r\\n at System.Activities.InArgument`1.TryPopulateValue(LocationEnvironment targetEnvironment, ActivityInstance activityInstance, ActivityExecutor executor)\\r\\n at System.Activities.ActivityInstance.ResolveArguments(ActivityExecutor executor, IDictionary`2 argumentValueOverrides, Location resultLocation, Int32 startIndex)\",\r\n \"CreationTime\": \"2026-06-24T09:14:08.117Z\",\r\n \"ReleaseName\": \"InvoiceNotifier\",\r\n \"Type\": \"Unattended\",\r\n \"InputArguments\": \"{}\",\r\n \"OutputArguments\": \"{}\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"RuntimeType\": \"Unattended\",\r\n \"RequiresUserInteraction\": false,\r\n \"ReleaseVersionId\": 10204,\r\n \"EntryPointPath\": \"Main.xaml\",\r\n \"OrganizationUnitId\": 28927,\r\n \"FolderKey\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"ProcessType\": \"Process\",\r\n \"ErrorCode\": \"Robot\",\r\n \"TraceId\": \"a1b2c3d4e5f64a7b8c9d0e1f2a3b4c5d\",\r\n \"Id\": 4019338\r\n }\r\n}\r\n" + }, + { + "argv": [ + "or", + "jobs", + "logs", + "a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"LogList\",\r\n \"Pagination\": {\r\n \"Returned\": 1,\r\n \"Limit\": 100,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Level\": \"Error\",\r\n \"TimeStamp\": \"2026-06-24T09:14:08.981Z\",\r\n \"Message\": \"InvoiceNotifier execution error: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run. (faulting activity: Assign 'Build Greeting' in Main.xaml)\",\r\n \"RobotName\": \"MOCK-HOST\",\r\n \"HostMachineName\": \"MOCK-HOST\",\r\n \"ProcessName\": \"InvoiceNotifier\",\r\n \"JobKey\": \"a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d\",\r\n \"RawMessage\": \"System.NotSupportedException: Expression Activity type 'VisualBasicValue`1' requires compilation in order to run.\"\r\n }\r\n ]\r\n}\r\n" + }, + { + "argv": [ + "or", + "folders", + "list" + ], + "match_mode": "subset", + "exit_code": 0, + "stdout": "{\r\n \"Result\": \"Success\",\r\n \"Code\": \"FolderList\",\r\n \"Pagination\": {\r\n \"Returned\": 2,\r\n \"Limit\": 50,\r\n \"Offset\": 0,\r\n \"HasMore\": false\r\n },\r\n \"Data\": [\r\n {\r\n \"Key\": \"dea5d8d6-ca39-4dbf-b4a5-3516bad38866\",\r\n \"Name\": \"automation@test.com's workspace\",\r\n \"Path\": \"automation@test.com's workspace\",\r\n \"Description\": \"\",\r\n \"Type\": \"Personal\",\r\n \"ParentKey\": \"\"\r\n },\r\n {\r\n \"Key\": \"1965a46b-db4e-469e-aaaa-7e0b379cb34d\",\r\n \"Name\": \"Shared\",\r\n \"Path\": \"Shared\",\r\n \"Description\": \"\",\r\n \"Type\": \"Standard\",\r\n \"ParentKey\": \"\"\r\n }\r\n ]\r\n}\r\n" + } + ], + "default": { + "exit_code": 0, + "stdout": "[]\n" + }, + "expected_calls": [ + { + "pattern": "or folders list", + "min": 1, + "description": "Agent enumerates folders to locate the Shared folder where the faulted job lives." + }, + { + "pattern": "or jobs get a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", + "min": 1, + "description": "Agent fetches the faulted job to read the 'requires compilation' error and the stack frame pointing at the Assign 'Build Greeting'." + }, + { + "pattern": "or jobs logs a1b2c3d4-e5f6-4a7b-8c9d-0e1f2a3b4c5d", + "min": 0, + "description": "OPTIONAL - error-level logs corroborate the error; the job-get Info already carries it." + } + ] +} diff --git a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/task.yaml b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/task.yaml index 0d316f7e1a..82224466c8 100644 --- a/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/task.yaml +++ b/tests/tasks/uipath-troubleshoot/runtime-exceptions/visualbasicvalue-smart-quotes/task.yaml @@ -23,22 +23,12 @@ run_limits: sandbox: template_sources: - - type: template_dir - path: ../../_shared/mock_template - type: template_dir path: process - - type: template_dir - path: data - mock_path_dirs: - - m - -pre_run: - # Seal the mock store before the agent starts: packs m/r/ into an opaque - # m/.store and removes m/r/, so the agent cannot read the recorded uip - # outputs and reach the diagnosis without investigating. - - command: "python m/seal" - timeout: 60 - fail_on_error: true + protected_mocks: + - tool: uip + fixture: data/uip-fixture.json + max_requests: 100 reference: file: RESOLUTION.md diff --git a/tests/tasks/uipath-troubleshoot/smoke-manifest-commands/task.yaml b/tests/tasks/uipath-troubleshoot/smoke-manifest-commands/task.yaml index 59e84fd6d2..1daecbdc64 100644 --- a/tests/tasks/uipath-troubleshoot/smoke-manifest-commands/task.yaml +++ b/tests/tasks/uipath-troubleshoot/smoke-manifest-commands/task.yaml @@ -1,6 +1,6 @@ task_id: skill-troubleshoot-smoke-manifest-commands description: > - Verify every uip subcommand mocked in any diagnostic scenario's manifest is + Verify every uip subcommand mocked in any diagnostic scenario's protected fixture is a real subcommand in the locally installed uip CLI. Catches fixture drift when CLI tools rename, remove, or reshape subcommands — the kind of breakage that previously slipped through because the mock dispatcher's permissive @@ -30,7 +30,7 @@ run_limits: # $env:TASK_DIR), so we resolve TASK_DIR through `python -c` which works the # same everywhere. pre_run: - # 1) Install every per-tool plugin the troubleshoot manifests reference. + # 1) Install every per-tool plugin the troubleshoot fixtures reference. # The bare `@uipath/cli` install in CI does NOT auto-list subcommands of # uninstalled tools in `--help` output, so the validator below would see # `or` / `is` / `maestro` / `docsai` / `traces` / `rpa` / `agent` /